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Abstract 


The  purpose  of  this  research  was  to  define  Systems  Engineering  applications  for 
Small  Business  Innovative  Research  (SBIR)  projects.  Specifically,  this  thesis  sought  to 
answer  five  research  questions  addressing  the  essential  elements  and  application  of 
Systems  Engineering  processes  within  the  SBIR  community.  Information  was  collected 
from  multiple  organizations  throughout  the  SBIR  community  to  support  this  research. 
The  research  identified  that  current  DoD  and  Air  Force  Systems  Engineering  Policy  do 
not  adequately  address  SBIR  projects  and  SE  processes  are  not  well  documented  within 
the  community.  This  research  identified  the  need  to  tailor  a  Systems  Engineering 
approach  for  SBIR  projects  as  overarching  policy  is  not  tailored  for  SBIR.  Results  from 
this  work  identified  the  applicable  SE  tasks  identified  in  Air  Force  policy.  The 
culmination  of  this  effort  defined  the  current  SE  tasks  applicable  in  the  SBIR  community 
as  well  the  overall  SE  rigor  being  applied  in  the  different  Systems  Engineering  Process 
areas  identified  in  DoD  and  Air  Force  Systems  Engineering  Policy. 
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I.  Introduction 


Background 

The  Air  Force  Small  Business  Innovation  Research  (SBIR)  program  is  vital 
element  of  the  Air  Force  Research  Laboratory  (AFRL)  contracts  portfolio  operated  under 
the  guidance  of  the  Air  Force  SBIR/Small  Business  Technology  Transfer  (STTR) 
Program  Manager  within  AFRL  at  Wright  Patterson  Air  Force  Base.  The  SBIR  program 
funds  early-stage  R&D  projects  at  small  technology  companies  that  support  a  Department 
of  Defense  (DoD)  need  and  have  the  potential  for  commercialization  in  private  sector 
and/or  military  markets.  The  DoD’s  SBIR  program  is  a  large  part  of  the  multibillion 
dollar  federal  SBIR  program  administered  by  twelve  federal  agencies  across  the  country 
[www.sbir.gov/about/about-sbir].  The  DoD  and  Air  Force  provide  top  level  Systems 
Engineering  guidance  and  policy  for  the  acquisition  community.  However  the  current 
guidance  and  policy  has  not  yet  been  tailored  specifically  for  the  SBIR  community. 

Challenges 

The  DoD  has  well  defined  system  engineering  processes  documented  in  the 
acquisition  5000  series  for  typical  acquisition  programs.  However  a  number  of 
challenges  exist  with  applying  Systems  Engineering  to  SBIR  projects  since  they  are 
unique  compared  to  typical  acquisition  programs.  They  are  managed  by  many  different 
small  businesses  that  may  or  may  not  have  an  organic  SE  capability.  Additionally  they 
vary  significantly  in  scope,  are  small  in  size,  short  in  project  length,  and  are  early 
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research  projects.  They  are  categorized  as  6.1  Basic  Research  or  6.2  Applied  Research 
projects  which  are  further  defined  in  AFRLI  61-204  Scientist  and  Engineer  Manuel. 
Topics  are  generated  across  the  Air  Force  by  Program  Executive  Officers,  Technolgy 
Directorates,  Air  Logistics  Centers  and  Test  Centers.  SBIR  projects  are  developed  in 
three  phases.  Phase  1  is  a  technical  feasibility  study  that  allocates  up  to  $  1 50k  and  9-12 
months.  Phase  II  is  concept  development  and  allocates  up  to  $1M  and  24  months.  There 
are  also  Critical  Manufacturing  SBIR  projects  that  are  allocated  up  to  $5M  for  Phase  II. 
Phase  III  is  the  commercialization  stage  [www.sbir.gov].  SBIR  projects  managed  by 
many  different  organizations  throughout  the  DoD.  Within  the  Air  Force,  SBIR  projects 
are  managed  by  AFRL  Technology  Directorates,  Test  Centers  and  Air  Logistics  Centers. 

Systems  engineering  is  a  technical  management  process  that  can  be  used  to  help 
ensure  that  projects  are  successfully  implemented  to  the  next  phase  of  development  if 
selected.  As  SBIR  projects  vary  considerably  in  scope,  are  managed  by  many  different 
organizations  within  the  government,  and  work  is  accomplished  by  varying  small 
businesses  it  presents  a  problem  of  ensuring  that  consistent  systems  engineering 
processes  are  being  applied  across  all  projects. 

Past  Research 

AFIT  past  research  efforts  have  helped  to  identify  areas  for  improvement  for 
applying  SE  to  the  S  &  T  community.  Most  notably  a  thesis  completed  by  AFIT  called 
“A  Tailored  SE  Framework  for  S  &  T  Projects”  developed  a  tailored  systems  engineering 
approach  for  typical  S  &  T  projects.  This  work  is  discussed  further  in  Chapter  II  as  part 
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of  the  literature  review.  SBIR  however  is  not  a  typical  S  &  T  project  and  AFRL  has 
expressed  interest  in  a  similar  project  that  would  define  the  SE  rigor  needed  for  a  SBIR 
project  and  provide  a  tailored  approach  for  implementing  SE  processes  into  their  SBIR 
projects.  Past  research  for  governing  this  material  is  covered  in  Chapter  II. 

Problem  Statement 

Systems  engineering  processes  are  not  fully  defined  in  policy  and  are  not  being 
implemented  consistently  and  to  adequate  levels  for  all  SBIR  projects.  SBIR  is  a  unique 
program  that  challenges  small  and  large  business  participants  to  work  together.  Small 
business  participants  may  or  may  not  have  SE  principals  as  defined  by  the  DoD 
incorporated  into  their  culture.  Identifying  the  adequate  level  of  SE  and  ensuring  it  is 
incorporated  in  a  consistent  manner  across  all  organizations  for  SBIR  projects  will  help 
to  ensure  projects  are  ready  to  proceed  to  the  next  phase  of  development.  This  will  aid  in 
the  future  transition  of  their  projects. 

Current  DoD  or  Air  Force  policy  does  not  specifically  define  SE  processes  for 
SBIR  projects.  AFRL  has  mapped  their  SE  policy  to  the  Defense  Acquisition  Guidebook 
(DAG)  in  AFRL  Instruction  61-104  Science  and  Technology  Systems  Engineering. 
However  application  of  Air  Force  Systems  Engineering  policy  has  yet  to  be  identified  for 
application  within  the  SBIR  program.  Current  Air  Force  policy  for  AFRL  programs  are 
outlined  in  AFRL  Instruction  61-104,  AFRL  Manuel  61-204  AFRL  Scientist  and 
Engineer  Manuel  and  AF  Instruction  63-1201  Life  Cycle  Systems  Engineering.  These 
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policies  are  in  alignment  with  the  DoD’s  Systems  Engineering  guidance  captured  in  the 
DoD  5000  series  and  the  DAG. 

Research  Focus 

The  focus  of  this  thesis  is  to  identify  how  current  systems  engineering  practices 
apply  for  SBIR  projects.  This  included  identifying  how  and  what  current  DoD  and  Air 
Force  SE  policy  apply  to  SBIR  project  during  Phase  I  and  II  and  how  to  best  tailor  the 
guidance  to  develop  a  solid  SE  approach  for  the  technical  management  of  the  project. 
Thus  this  thesis  focuses  on  implementation  of  early  systems  engineering  processes  for 
Phase  I  and  II  SBIR  projects.  Without  a  solid  SE  approach  SBIR  projects  are  at  risk  to 
fail.  Good  SE  processes  will  help  to  ensure  projects  better  prepared  for  proceeding  to 
their  next  phase  of  development  while  adequately  managing  technical  risk. 

Methodology 

Preliminary  research  included  identifying  relevant  SBIR  documentation,  past  case 
studies  and  previous  work.  It  was  quickly  identified  that  there  was  insufficient  SBIR 
documentation  to  support  this  approach.  Very  little  Systems  Engineering  documentation 
was  found  to  be  associated  with  SBIR  projects  and  varied  among  organizations.  Thus  it 
became  essential  to  conduct  interviews  to  gather  the  information  needed.  Then  interview 
and  literature  review  data  could  be  analyzed  using  a  triangulation  approach  to  identify  the 
relevant  Systems  Engineering  application  of  principles. 
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Assumptions/Limitations 

No  case  studies  that  focused  on  direct  application  of  systems  engineering  on  SBIR 
projects  were  found.  Since  organizations  managing  SBIR  projects  are  geographically 
separated  it  is  not  feasible  to  gather  data  from  enough  organizations  to  have  valid  data 
that  represents  all  SBIR  projects.  Therefore,  this  study  is  based  on  representative 
sampling. 

Implications 

Though  this  project  focuses  specifically  on  SBIR  projects,  findings  will  likely  be 
applicable  to  similar  S  &  T  projects.  Projects  in  early  developments  will  have  many 
similar  attributes  to  the  SBIR  projects  analyzed  in  this  research.  This  work  could  be  used 
to  guide  a  tailored  SE  approach  for  similar  projects/programs. 

Summary 

This  chapter  provided  an  overview  of  research.  Chapter  II  will  review  relevant 
literature.  Chapter  III  will  provide  an  in  depth  look  at  the  methodology.  Chapter  IV  will 
analyze  data  for  this  research.  Chapter  V  will  provide  results  and  conclusions  for  this 
research. 
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II.  Literature  Review 


Chapter  Overview 

The  purpose  of  this  chapter  is  to  review  past  work  accomplished  on  Systems 
Engineering  in  AFRL  and  identify  current  SE  policy  as  it  pertains  to  the  S  &T 
Community.  The  Department  of  Defense  has  published  the  Defense  Acquisition  Guide 
and  the  DoD  5000  series  to  identify  SE  processes.  The  Air  Force  has  published  Air 
Force  Instruction  (AFI)  63-1201”  Life  Cycle  Systems  Engineering”  for  the  acquisition 
community.  Additionally,  Air  Force  Material  Command  developed  the  Air  Force 
Systems  Engineering  Assessment  Model  (AF  SEAM)  for  assessment  of  Air  Force 
programs.  AF  SEAM  is  a  very  useful  SE  assessment  tool  for  typical  Air  Force 
acquisition  programs.  Differences  between  the  DAG,  AFI  63-1201  and  AF  SEAM  can 
be  confusing  since  they  vary  in  terminology.  The  below  graphic  identifies  the  SE 
processes  identified  in  each  document. 
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Table  1:  SE  Processes  (AF  SEAM,  Sept  2010) 


AF  SEAM 

Defense  Acquisition  Guide 

AFI  63-1201 

Requirements 

Reqts  Analysis,  Reqts  Mgmt. 
Stakeholder  Reqts  Definition 

Reqt  Dev  &  Mgmt.  & 
Architecture 

Design 

Architectural  Design, 
Integration  &  Interface 
Mgmt 

Design  &  Interface  Mgmt 

Verification  &  Validation 

A  erification  &  Validation 

Test  &  Evaluation, 
Verification  &  Validation 

Manufacturing 

Implementation 

Design 

Transition,  Fielding,  & 
Sustainment 

Transition 

Design 

Project  Planning 

Technical  Planning 

Planning 

Configuration 

Management 

CM,  Data  Mgmt.  Technical 
Data  Mgmt 

Configuration  Mgmt.  Data 
Mgmt 

Risk  Management 

Risk  Mgmt 

Integrated  Risk  Management 

Technical  Mgmt  & 
Control  (PMC) 

Technical  Assessment 

Technical  Reviews  & 
Measurements 

Decision  Analysis 

Decision  Analysis 

Decision  Analysis 

Defense  Acquisition  Guidebook 

The  Defense  Acquisition  Guidebook  (DAG)  is  designed  to  complement  DoD 
policies  identified  in  DoD  Directive  5000.01  “The  Defense  Acquisition  System”  and 
DoD  Instruction  5000.02  “Operation  of  Defense  Acquisition  System”.  Chapter  4  of  the 
DAG  covers  Systems  Engineering.  It  “covers  the  system  design  issues  facing  a  program 
manager,  and  details  the  Systems  Engineering  processes  that  aid  the  program  manager  in 
designing  an  integrated  system  that  results  in  a  balanced  capability  solution”  [DAG, 
2012]. 
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It  defines  Systems  Engineering  as: 

an  interdisciplinary  approach  and  process  encompassing  the  entire  technical 
effort  to  evolve,  verify  and  sustain  an  integrated  and  total  life  cycle  balanced  set 
of  system,  people,  and  process  solutions  that  satisfy  customer  needs.  Systems 
Engineering  is  the  integrating  mechanism  for  the  technical  and  technical 
management  efforts  related  to  the  concept  analysis,  materiel  solution  analysis, 
engineering  and  manufacturing  development,  production  and  deployment, 
operations  and  support,  disposal  of,  and  user  training  for  systems  and  their  life 
cycle  processes  [DAG,  2012]. 

The  DAG  section  4. 1.3. 1.1  discusses  early  Systems  Engineering  and  emphasizes  the 
importance  of  early  SE  during  technology  development.  The  DAG  also  defines  the  role 
of  the  Program  Manager  and  Chief  Engineer  illustrated  in  figure  2  below.  The  DAG  also 
separates  the  above  16  SE  processes  into  two  areas  shown  in  figure  3. 


Table  2:  DAG  Processes  and  Roles  of  the  PM  and  SE  (DAG  Table  4.1.1T1,  2012) 


Life-cycle  Processes 

Program 

Manager 

Chief/ 

Systems 

Engineer 

Stakeholder  Management 

Primary 

Support 

Technical  Planning 

Support 

Primary 

Decision  Analysis 

Primary 

Support 

Technical  Assessment  (Includes  Program 
Status:  Technical  Progress,  Schedule  &  Cost 
Management) 

Shared 

Shared 

Configuration  Management 

Primary 

Support 

Data  Management 

Primary 

Support 

Requirements  Management 

Support 

Primary 

Contract  Management 

Primary 

Support 

Requirements  Allah’s  is 

Support 

Primary 

Architecture  Design 

Support 

Primary 

Implementation 

Support 

Primary 

Risk  Management 

Primary 

Support 

Interface  Management 

Support 

Primary 

Integration 

Support 

Primary 

Verification 

Support 

Primary 

Validation 

Shared 

Shared 
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Table  3:  DAG  SE  Processes  (DAG  Table  4.2.3.T1,  2012) 


Technical  Management  Processes 

Technical  Processes 

Decision  Analysis 

Stakeholders  Requirements  Definition 

Technical  Planning 

Requirements  Analysis 

Technical  Assessment 

Architectural  Design 

Requirements  Management 

Implementation 

Risk  Management 

Integration 

Configuration  Management 

Verification 

Technical  Data  Management 

Validation 

Interface  Management 

Transition 

AFRLI  61-104  attempts  to  translate  these  processes  from  the  DAG  for  the  science  and 
technology  community.  Also  section  4. 3. 2. 3  of  the  DAG  identifies  the  following  SE 
tasks  relevant  to  the  S  &  T  community: 

•  Key  Systems  Engineering  Activities  During  Technology  Development 

•  Interpret  User  Needs;  Analyze  Operational  Capability  and 
Environmental  Constraints 

•  Develop  System  Performance  (and  Constraints)  Specifications  and 
Enabling/Critical  Technologies  and  Prototypes  Verification  Plan 

•  Develop  Functional  Definitions  for  Enabling/Critical  Technologies/Prototypes 
and  Associated  Verification  Plan 

•  Decompose  Functional  Definitions  into  Critical  Component  Definition  and 
Technology  Verification  Plan 

•  Design/Develop  System  Concepts,  i.e.,  Enabling/Critical  Technologies; 
Update  Constraints  and  Cost/Risk  Drivers 

•  Demonstrate  Enabling/Critical  Technology  Components  Versus  Plan 

•  Demonstrate  System  and  Prototype  Functionality  Versus  Plan 

•  Demonstrate/Model  the  Integrated  System  Versus  the  Performance 
Specification 

•  Demonstrate  and  Validate  the  System  Concepts  and  Technology  Maturity 
Versus  Defined  User  Needs 

•  Transition  to  Integrated  System  Design 

•  Interpret  User  Needs,  Refine  System  Perfonnance  Specifications  and 
Environmental  Constraints 
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•  Develop  System  Functional  Specifications  and  Verification  Plan  to  Evolve 
System  Functional  Baseline 

•  Evolve  Functional  Performance  Specifications  into  System  Allocated  Baseline 
The  DAG  also  identifies  the  following  for  SBIR: 

2.2.10.1.  Small  Business  Innovation  Research  (SBIR)  Technologies 

Consistent  with  the  direction  of  DoD  Instruction  5000.02,  the  program  manager 
(PM)  should  prepare  a  Technology  Development  Strategy  (TDS)  that 
appropriately  uses  the  SBIR  program  to  develop  needed  technologies,  includes 
the  use  of  technologies  developed  under  the  SBIR  program,  and  gives  fair 
consideration  to  successful  SBIR  technologies.  During  TDS  preparation,  the  PM 
should  ensure  that  the  strategy  addresses  transition  of  relevant  SBIR  technologies 
and  includes  budgeting  of  follow-on  funds  for  test,  evaluation,  and  integration,  as 
needed,  to  achieve  the  desired  technological  maturity.  In  addition,  the  PM  should 
consider  SBIR  technologies  as  candidates  for  incremental  and  block  system 
improvement  initiatives  as  well  as  to  address  competitive  prototyping 
requirements,  particularly  at  the  subsystems  and  component  levels.  To  effectively 
leverage  SBIR,  the  PM  review  and  ensure  compliance  with  DoD  SBIR  Phase  III 
policy  guidance  and  should  engage  their  program  office,  Program  Executive 
Office,  systems  command,  product  center,  or  DoD  Component  SBIR  program 
coordinator  for  assistance.  The  PM  should  also  consult  the  DoD  SBIR  program 
Web  site  for  online  resources  and  information  including  a  program  description, 
database  of  past  awards  and  key  points  of  contracts. 

2.3.10.1.3.  Small  Business  Innovation  Research  (SBIR)  Consideration 

Consistent  with  the  direction  of  DoD  Instruction  5000.02,  the  PM  should  include 
SBIR  and  give  fair  consideration  to  successful  SBIR-fimded  technologies  in 
Acquisition  Strategy  planning.  Note  that  SBIR  follow-on  development  and 
acquisition  (Phase  III,  not  funded  with  the  SBIR  set-aside  budget)  may  be  able  to 
be  pursued  on  a  sole-source  basis  without  further  competition.  Competition  for 
Phase  I  and  Phase  II  awards  (contracts  funded  by  the  SBIR  set-aside  budget) 
satisfies  all  statutory  competition  requirements.  SBIR  Phase  III  contract  awards 
have  SBIR  status  and  thus  must  be  accorded  SBIR  data  rights.  SBIR  Phase  III 
work  may  be  pursued  directly  through  Phase  III  contracts  or  encouraged  through 
subcontracts  via  incentives.  To  effectively  leverage  SBIR,  the  PM  review  and 
ensure  compliance  with  DoD  SBIR  Phase  III  policy  guidance  and  should  engage 
their  program  office,  Program  Executive  Office  (PEO),  systems  command, 
product  center,  or  Component  SBIR  program  coordinator  for  assistance. 
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Air  Force  Instruction  63-1201  Life  Cycle  Systems  Engineering 

AFI  63-120  identifies  SE  as  encompassing  “the  entire  set  of  scientific,  technical, 
and  managerial  efforts  needed  to  conceive,  evolve,  verify,  deploy,  support,  and  sustain  a 
robust  product,  platform,  system,  or  integrated  system-of-systems  (SoS)  capability  to 
meet  user  needs”.  It  currently  defines  12  SE  processes  that  were  shown  earlier  in  Table 
1 .  It  also  defines  SE  responsibilities  for  the  program  manager  and  engineers.  It  however 
does  not  specifically  address  SBIR  projects. 

AFI  63-1201  is  currently  under  revision  and  is  projected  to  better  align  with  AF 
SEAM  and  the  DAG.  The  revised  draft  is  projected  to  align  with  AF  SEAM’s  10  SE 
processes.  The  DAG  defines  16  SE  processes  as  also  illustrated  in  Table  1.  This 
disconnect  in  policy  can  be  confusing  for  project  managers  and  engineers  trying  to 
decipher  how  policy  applies  to  their  projects  and  how  best  to  develop  a  solid  SE 
approach.  The  revised  version  of  AFI  63-1201  should  reduce  this  significantly  by 
eliminating  the  current  differences  in  the  AFI.  The  author  noted  these  changes  in  his 
review  of  the  2011  draft  versions  of  the  updated  AFI  63-120 1 .  Discussions  to  make  the 
new  version  AFI62-101  have  been  held  but  no  decision  has  yet  been  made  whether  the 
updated  version  will  be  AFI  63-1201  or  62-101. 

With  these  changes  coming  to  the  AFI  organizations  should  be  considering  these 
changes  for  future  policy  updates.  Also,  they  should  be  ready  to  ensure  that  these  SE 
processes  are  being  executed  properly  once  the  new  policy  is  published. 
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Air  Force  Systems  Engineering  Assessment  Model  (SEAM) 

The  primary  purpose  of  AF  SEAM  is  to  promote  the  application  and  use  of 
standard  SE  processes  across  the  AF  and  to  improve  the  perfonnance  of  these  processes 
through  Continuous  Process  hnprovement  [AF  SEAM,  September  2010].  AF  SEAM  is 
not  yet  mandated  across  the  Air  Force  however  it  is  used  as  a  reporting  tool  in  some  AF 
communities.  It  is  also  projected  that  the  revised  AFI  63-1201  will  align  with  AF  SEAM 
AF  SEAM  identifies  “ten  AF  standard  SE  process  areas”  and  lists  associated  goals  for 
each.  Specific  practices  and  generic  practices  are  identified  for  each  area.  Those  areas 
are  seen  below  in  Table  4  along  with  the  number  of  practices  for  each  area. 

Table  4:  AF  SEAM  SE  Total  Practices  (AF  SEAM,  Sept  2008) 


Specific 

Generic 

Total 

Process  Area 

Goals 

Practices 

Practices 

Practices 

Configuration  Mgmt 

3 

8 

7 

16 

Decision  Analysis 

1 

5 

7 

12 

Design 

3 

14 

7 

21 

Manufacturing 

4 

12 

7 

ie 

Project  Planning 

3 

15 

7 

22 

Requirements 

4 

13 

7 

21 

Risk  Mgmt 

3 

7 

7 

14 

Trans.  Fielding.  &  Sus 

4 

15 

7 

22 

Tech  Mgmt  &  Control 

4 

15 

7 

18 

Verification  &  Validation 

5 

16 

7 

23 

Total 

34 

120 

70 

190 

As  seen  above  AF  SEAM  identifies  190  total  practices.  This  suggests  a  significant  SE 
effort  for  any  program  to  implement  AF  SEAM.  There  are  three  different  training 
modules  developed  to  support  AF  SEAM  describe  in  further  detail  in  section  7  of  AF 
SEAM.  My  experience  as  a  SE  instructor  has  taught  me  that  this  is  a  good  rigorous  tool 


12 


for  a  major  acquisition  programs.  However  it  must  be  tailored  to  a  smaller  scope  to  be 
value  added  for  a  smaller  project  since  it  requires  a  large  manpower  effort.  This  is 
significant  for  SBIR  projects  since  their  limited  scope  and  resources  require  an  even  more 
tailored  approach  to  be  value  added.  AF  SEAM  was  designed  to  facilitate  use  tailoring. 
Not  applicable  tasks  can  be  coded  N/A  and  not  be  assessed.  Generic  practices  can  also 
be  omitted.  This  ability  to  tailor  it  to  a  specific  project  still  requires  a  considerable  effort 
since  so  many  of  the  task  may  not  apply  for  S  &  T  projects.  This  is  even  more  so  for 
SBIR  projects. 

Additionally,  discussions  with  AFRL  Plans  and  Programs  quickly  identified  that 
AF  SEAM  is  a  very  rigorous  tool  for  implementing  SE  that  is  not  tailored  to  an 
appropriate  level  for  AFRL  projects.  It  is  not  tailored  for  the  S&T  community.  In  its 
current  configuration,  as  many  of  the  190  SE  tasks  may  or  may  not  apply  given  the 
attributes  of  the  AFRL  project  or  program  implementing  this  “as  is”  does  not  make  sense 
for  the  SBIR  community  due  to  the  uniqueness  of  their  projects,  limited  resources  and 
limited  value  added  to  the  project.  A  more  tailored  approach  is  required.  Analysis  of 
interview  results  in  chapter  IV  for  each  SE  process  will  identify  what  SE  tasks  are  being 
implemented  and  what  are  applicable  for  the  SBIR  community. 
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Air  Force  Research  Lab  Policy 


The  Air  Force  Research  Labs  have  two  main  documents  that  provide  guidance  for 
Systems  Engineering.  The  first  is  AFRL  Instruction  61-104  which  specifically  addresses 
Systems  Engineering  in  the  S&T  environment.  The  second  is  AFRL  61-204  Manual  for 
Scientist  and  Engineers. 


AFRL  Instruction  61-104  S&T  Systems  Engineering 

AFRL  Instruction  61-104  “Science  and  Technology  Systems  Engineering”  provides 
SE  guidance  for  all  of  the  AFRL  community.  It  is  in  alignment  with  DoD  and  Air  Force 
policy  but  tailors  it  to  the  S&T  community.  It  identifies  Eight  Systems  Engineering  Key 
Questions  to  guide  and  assess  the  SE  health  of  a  project.  The  questions  are: 

1 .  Who  is  your  customer? 

2.  What  are  the  customer’s  requirements? 

3.  How  will  you  demonstrate  you  have  met  the  requirements? 

4.  What  are  the  technology  options? 

5.  Which  is  the  best  approach? 

6.  What  are  the  risks  to  developing  the  selected  technology? 

7.  How  will  you  structure  your  program  to  meet  requirements  and  mitigate  risk? 

8.  What  is  your  business-based  transition  plan  that  meets  customer  approval? 

AFRLI  61-104  maps  these  questions  back  to  the  Systems  Engineering  “Vee”  identified 
below  from  the  Defense  Acquisition  Guidebook. 
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Figure  Al.l.  The  Systems  Engineering  Yee 


Technical  Management  Processes 

Decision  Analysis  Tech  Planning  Tech  Assessment 
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Figure  1:  AFRL  Processes  Mapped  to  the  SE  Vee  (AFRLI  61-104,  2008) 


AFRLI  61-104  Attachment  1  also  maps  the  Eight  SE  Key  Questions  back  to  the  DAG  SE 
areas  which  is  illustrated  below  in  Table  5. 
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Table  5:  AFRL  8  SE  Key  Questions  Mapped  to  the  DAG 


Question 

Design 

Analysis 

Tech 

Planning 

Tech 

Assess 

ment 

Req 

Mgmt 

Risk 

Mgmt 

Conf. 

Mgmt 

Data 

Mgmt 

Interface 

Mgmt 

Req't 

Dev 

Logical 

Analysis 

Design 

Solution 

Implem 

entation 

Integr 

ation 

Verific 

ation 

Valid 

ation 

Trans 

ition 

1 .  Who  is  your 
customer'1 

X 

X 

2.  What  are  the 

customer's 

requirements? 

X 

X 

X 

X 

X 

X 

3.  How  will  you 
demonstrate  you  have 
met  the  requirements? 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

4.  What  are  the 
technology  options? 

X 

X 

X 

5.  Which  is  the  best 
approach? 

X 

X 

X 

X 

X 

6.  What  are  the  risks  to 
developing  the  selected 
technology? 

X 

X 

X 

7.  How  will  you 
structure  your  program 
to  meet  requirements  and 
mitigate  risk? 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

8.  What  is  your 
business-based  transition 
plan  that  meets  customer 

X 

X 

X 

X 

AFRLI  61-104  Attachment  1  provides  a  question  and  answer  matrix  for  the  Eight  SE  Key 
Questions.  It  breaks  the  Eight  SE  Key  Questions  down  to  further  detail,  identifies  “What 
the  Program  Manager  should  know  about  his  or  her  program”  and  defines  the  color 
assessment  basis.  It  also  identifies  that  “Use  of  the  key  questions  during  reviews  of  basic 
research  programs  is  optional.”  It  does  this  separately  for  6.1  Basic  Research,  6.2 
Applied  Research,  6.3  Advanced  Technology  Development,  Advanced  Technology 
Demonstration  and  Manufacturing  Technology. 

Also  in  Attachment  1,  AFRL  translates  the  16  SE  DAG  processes  for  the  AFRL 
community.  It  defines  the  DAG  process,  defines  the  process  for  AFRL  and  explains  the 
importance.  Review  of  this  attachment  identified  that  it  is  tailored  for  the  typical  AFRL 
program  and  not  tailored  for  SBIR  projects  specifically. 
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Additionally  AFRLI  61-104  is  currently  being  revised.  The  AFRL  Systems 
Engineering  Council  is  currently  reviewing  the  document.  The  new  version  is  much 
shorter  and  uses  an  AFRL  Systems  Engineering  Guidebook  to  companion  the  document. 
The  guidebook  details  how  to  implement  SE  processes  into  a  program  or  project.  Both 
the  instruction  and  guidebook  define  the  S&T  SE  Process  in  the  below  illustration: 


Figure  1.0  The  S&T  SE  Process. 


Customer 

Requirements 


Technology ' 
Alternatives 


Technology 

Demonstration 


Figure  2:  AFRL  S&T  SE  Process  (Draft  AFRL  SE  Guidebook,  2012) 

This  is  similar  to  the  SE  streamlined  process  identified  in  the  case  study  review.  This 
process  was  successfully  implemented  in  past  case  studies.  Each  step  is  explained  in 
further  detail  to  identify  what  SE  tasks  should  be  perfonned.  The  guidebook  also 
indentifies  the  Eight  SE  Key  Questions  and  explains  what  should  be  done  for  each  as 
indentified  in  the  current  instruction. 
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AFRL  Manuel  61-204  AFRL  Scientist  and  Engineer  Manuel 

The  intent  of  this  manual  is  to  “enhance  DoD  Directives  and  Air  Force 
Instructions  by  placing  actions  into  a  chronological  process  flow  specifically  developed 
for  the  management  of  AFRL  Work  Units”.  Thus  the  manual  provides  guidance  on  how 
to  technically  manage  work  units  within  AFRL. 

The  manual  identifies  SBIR  as  a  three  phase  congressionally  mandated  program 
“established  to  stimulate  technological  innovation,  use  small  businesses  to  meet  federal 
R&D  needs,  increase  innovative,  private  sector  R&D  commercialization,  and  to 
encourage  minority  and  disadvantaged  persons  to  participate  in  technological  innovation” 
[AFRLM  62-204,  2003].  It  identifies  the  different  phases,  funding  and  duration.  It  also 
defines  the  SBIR  schedule  from  initial  topic  call  to  phase  completions. 

The  manual  defines  6.1  Basic  Research  as  a  “systematic  study  directed  toward 
greater  knowledge  or  understanding  of  the  fundamental  aspects  of  phenomena  and  of 
observable  facts  without  specific  applications  towards  processes  or  products  in  mind.”  It 
also  defines  6.2  Applied  Research  as  a  “systematic  study  to  understand  the  means  to  meet 
a  recognized  and  specific  national  security  requirement”  [AFRLM  62-204,  2003].  More 
information  for  both  research  categories  can  be  found  in  the  manual. 
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Policy  Summary 

Review  of  Systems  Engineering  policy  within  the  DoD,  Air  Force  and  AFRL  has 
identified  very  rigorous  defined  SE  guidance  and  instruction.  AFRL  has  done  a  good  job 
tailoring  their  policy  to  be  in  alignment  with  higher  level  guidance.  Additionally  AFRL’s 
System  Engineering  Council  continues  to  be  proactive  in  the  development  of  the  new  SE 
Guidebook  for  better  implementing  good  SE  processes  within  AFRL.  However  current 
policy  at  all  the  reviewed  levels  is  not  specific  enough  for  the  SBIR  community.  Since 
SBIR  is  unique  in  many  aspects  as  previously  discussed  SE  guidance  needs  to  be  better 
defined  and  tailored  for  the  SBIR  community  to  ensure  it  is  being  implemented 
successfully.  SBIR  projects  are  at  risks  to  not  incorporate  adequate  SE  processes  without 
better  guidance.  Additionally  any  future  efforts  to  tailor  SE  processes  for  the  SBIR 
community  should  be  in  alignment  with  AF  SEAM  processes  as  the  revised  AFI  63-1201 
is  projected  to  align  with  AF  SEAM. 

Past  Research 

Several  efforts  including  AFIT  graduate  thesis  work  and  past  studies  have  been 
accomplished  to  analyze  Systems  Engineering  efforts  within  AFRL.  The  author 
identified  most  notably  a  past  research  project  “A  Tailored  SE  Framework  for  S  &  T 
Projects”  authored  by  Maj  Pitzer,  Maj  Behm  and  Jane  White  that  captured  the  Systems 
Engineering  tasks  and  rigor  applicable  for  typical  AFRL  projects.  They  developed  a  tool 
called  the  “Systems  Engineering  Tailoring  tool  for  Science  &  Technology  Projects”  that 
defined  projects  by  6  parameters: 


19 


1.  RDT&E  Category  (6.1,  6.2  or  6.3) 

2.  Project  Budget  (less  than  $500k,  $500k-$2M,  greater  than  $2M) 

3 .  Core  Process  (CP- 1 ,2  or  3) 

4.  Technology  Readiness  Level  ( 1  thru  9) 

5.  Integration  Level  (subsystem,  system  or  mission) 

6.  Requirements  Maturity  (Technology  Push  or  Requirements  Pull) 

The  tool  then  outputted  what  SE  best  practices  (mapped  from  the  16  DAG  processes)  that 
would  apply  to  that  project/program.  This  tool  is  notably  similar  to  AF  SEAM  however 
it  tailors  the  tasks  for  a  project  based  on  the  stated  parameters.  The  SETT  tool  provides  a 
good  initial  baseline  however  SBIR  projects  are  unique  as  previously  identified. 
Preliminary  analysis  of  the  SETT  Tool  identified  that  it  also  does  not  tailor  specifically  to 
the  SBIR  community.  Like  AF  SEAM,  SETT  identifies  many  tasks  that  may  not  be 
applicable  to  a  specific  SBIR  project  due  to  its  unique  attributes.  .  Implementing  a 
process  or  tool  that  is  not  tailored  to  the  appropriate  level  risks  creation  of  non  value 
added  work  and  can  drain  valuable  resources  from  a  project.  Both  the  SETT  Tool  and 
AF  SEAM  are  good  baselines  to  consider  when  identifying  what  SE  tasks  may  apply  to 
SBIR  projects.  Typical  parameters  for  a  SBIR  project  inputted  into  the  SETT  Tool  to 
establish  a  baseline  are  illustrated  in  Appendix  3. 

Additionally  the  author  identified  four  notable  AFRL  studies  that  were  significant  for 
this  research: 

1 .  High  Energy  Laser  On  a  Large  Tactical  Platfonn  (HELLTP) 

2.  Deployed  Base  Energy  Alternatives  Report 

3.  Company  Grade  Officer  Initiative  Program 

4.  AFRL  Transformational  Activities  in  Systems  Engineering  (TASE)  Assessment 

Phase  Final  Report  Findings  from  2006 
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The  first  two  studies  focused  on  the  successful  tailoring  and  streamlining  of  SE  efforts  on 
two  larger  AFRL  projects.  The  third  study  listed  focused  on  tailoring  and  streamlining 
SE  efforts  to  smaller  projects  within  AFRL  as  part  of  CGOIP.  This  study  was  very 
interesting  since  the  projects  were  being  managed  by  CGOs  with  limited  SE 
backgrounds.  And  like  the  first  two  studies  listed,  CGOIP  was  also  very  successful  in 
implementing  good  SE  processes  into  their  projects  using  a  streamlined  SE  approach. 

The  last  study  focused  on  making  AFRL  research  programs  more  effective  and  efficient, 
and  improving  the  transition  of  technology  to  the  warfighting  community  through  the  use 
of  good  SE  processes.  A  number  of  very  interesting  findings  were  documented  in  this 
report. 

The  studies  selected  focused  on  S&T  projects  that  successful  implemented  SE 
processes.  The  goal  was  to  establish  a  successful  baseline  from  historical  examples  that 
define  the  SE  rigor  needed  in  the  S&T  community.  The  case  studies  focused  on  projects 
that  fonned  a  multi-disciplinary  team  and  implemented  a  tailored  streamlined  SE 
approach  for  their  projects.  This  approach  proved  to  be  very  successful  in  implementing 
good  SE  processes  for  the  projects.  This  approach  could  be  very  beneficial  for  SBIR 
projects  if  tailored  to  the  appropriate  level.  The  four  studies  identified  were  analyzed  for 
SE  artifacts  that  contributed  to  their  success. 
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High  Energy  Laser  On  a  Large  Tactical  Platform  (HELLTP) 

The  initial  phase  of  this  project  implemented  the  Air  Force’s  Integrated  Product 
and  Process  Development  (IPPD)  process.  The  project  was  considered  a  Multi¬ 
directorate  SE  Initiative.  It  had  three  main  objectives: 

1 .  Apply  the  IPPD  process  to  the  selected  High  Energy  Laser  on  a  Large  Tactical 
Platform  (HELLTP)  problem  across  multiple  directorates,  with  “customer” 
involvement. 

2.  Assess  the  tools  and  process  in  the  course  of  executing  the  program. 

3.  Capture  lessons  learned  with  comments  and  recommendations  for  going  forward 
in  the  Phase  II  program. 

The  project  was  able  to  establish  a  successful  team  framework  throughout  the  IPPD 
process.  As  a  result  the  team  was  able  to  tailor  their  SE  approach  for  the  project.  The 
below  figure  illustrates  their  approach. 
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Figure  3:  HELLTP  Top  Level  IPPD  Model 
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The  team  identified  the  need  to  tailor  their  approach  as  well  as  tailor  the  SE  tools  for  the 
project.  They  classified  their  tools  into  four  classifications:  Preliminary  SE, 
Requirements  Management  and  Evaluation,  System  Architecture  Tools,  and  Modeling 
and  Simulation.  From  there  they  were  able  to  select  the  tailored  tools  needed  for  their 
project.  Additionally,  the  team  relied  on  a  SE  contractor  to  provide  just  in  time  training 
to  the  team.  As  part  of  the  IPPD  process  the  team  conducted  reoccurring  meetings  to 
access  project  status  and  progress. 

Deployed  Base  Energy  Alternatives  Report 

The  study  focused  on  application  of  Systems  Engineering  principles  to  improve 
technology  investment  outcomes.  The  project  elected  to  use  a  contractor  developed 
method  called  Systems  Engineering  Tailored  for  Science  and  Technology  (SETFST). 
The  following  steps  were  implemented: 

1.  Establish  the  study  team  (IPT)  and  define  the  overall  program  objectives. 

2.  Define  Desirements  with  team. 

3.  Generate  alternatives. 

4.  Score  alternatives. 

5.  Exercise  a  value  analysis  model,  and  prioritize  the  alternatives. 

Formal  team  meetings  were  held  with  the  IPT  to  review  project  status.  All  key 
stakeholders  were  represented  with  membership  on  the  IPT.  The  team  developed  a 
defined  technical  approach  for  the  project.  As  a  result  the  team  successfully 
implemented  the  SE  processes  identified  in  the  SETFST  method  selected. 
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Key  findings: 

Review  of  the  report  clearly  identifies  that  success  of  the  project  was  in  direct 
relationship  with  the  success  of  the  IPT.  The  team  construct  ensured  that  they  had  the 
right  mix  of  expertise  for  the  project  as  well  as  team  members  with  the  expertise  in 
Systems  Engineering  to  successfully  implement  the  process.  This  technical  approach  to 
the  problem  was  very  successful.  The  team  implemented  the  SERFST  method  which  is 
consistent  with  DoD  and  AF  policy.  The  SETST  method  is  similar  to  the  streamline  S&T 
process  outlined  in  current  AFRL  policy  and  is  being  incorporated  into  the  AFRL  SE 
Guide  to  companion  the  revised  AFRLI  61-104. 

Company  Grade  Officer  Initiative  Program  (CGOIP) 

AFRL/RX  piloted  the  Company  Grade  Officer  Initiative  Program  (CGOIP)  in  an 
effort  to  streamline  and  tailor  the  SE  effort  for  their  projects.  The  projects  were  managed 
by  CGO’s  that  had  minimal  experience  in  SE.  These  projects  had  an  emphasis  on 
transition  their  projects.  The  projects  were  successful  in  implementing  SE  into  the 
projects  by  using  the  5  step  Streamlined  SE  Approach  and  Proposal  Checklist  illustrated 
below. 
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Streamlined  Systems  Engineering  Approach 


Step  1  Step  2  Step  3  Step  4  Step  5 
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-  Reliability 

-  Supportability 

□  Prioritize  &  Validate 
with  customer 

□  Team 
brainstorm 
different 
solution 
approaches 

□  Analysis  of 
Alternatives 
-  Compare  Alt. 
Solutions  with 
Reqts  &  Exit  Crit 

□  Tech  Readiness 
Assessment 

□  Manufacturing 
Readiness 
Assessment 

□  Risk  Analysis 

□  Customer 
Approval  of  Soln 


□  USAF  Problem 
/Goal 

□  Customer(s) 

□  SE  Team 
Members 

□  List  of  Prioritized 
Reqts  &  Exit 
Criteria 

□  Description  of 
Alternative 
Solutions 

□  TRAand  MRA 

□  Risk  Assessment 

□  Value  Analysis 
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Figure  4:  CGOIP  Streamlined  SE  Approach 


CGO  IP  Proposal  Checklist 


□  USAF  Problem  /  Goal 

□  Customer(s)  and  User(s) 

□  IPT  Members 

□  List  of  Requirements,  KPPs,  and  S&T  Exit  Criteria 

-  Objectives  and  Thresholds 


Program  Requirements 


□  Alternative  Solution  Approaches 

□  TRA  and  MRA 

□  Risk  Assessment 

-  Identification  and  Mitigation 

□  Value  Analysis  for  Selecting  Best  Approach 


Evaluation  of  Alternatives 


□ 

In-house  Work  Tasks 

Program  Plan 

□ 

Materials  or  Manufacturing  Technology  Related 
-  Rationale  for  why  is  RX  doing  this 

□ 

Test  Plan 

□ 

Proposed  Cost  &  Spend  Plan 

□ 

Schedule  with  Major  Milestones 

□ 

Technology  Transition  Strategy 

Figure  5:  CGOIP  Proposal  Checklist 
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Following  this  streamlined  approach  enabled  the  teams  to  successfully  implement  SE  into 
their  processes.  As  a  result  their  programs  were  successful.  This  method  was  similar  to 
the  streamlined  SE  process  used  for  case  study  2. 


AFRL  Transformational  Activities  in  Systems  Engineering  (TASE)  Assessment 
Phase  Final  Report  Findings  from  2006 

The  goal  of  the  project  was  to  “make  AFRL  research  programs  more  effective  and 
efficient  (improve  S&T  program  performance),  and  improve  the  transition  of  technology  to 
the  warfighting  community  (improve  technology  transition)”.  The  team  found  a  number  of 
interesting  findings  as  seen  below. 

Main  trends  discovered: 

•  AFRL  program  personnel  already  have  guidance  on  sound  Systems  Engineering 
practices  (AFRLI  61-104,  Science  and  Technology  (S&T)  Systems  Engineering 
(SE)  Initiative).  However,  this  guidance  has  some  shortcomings.  AFRL  should 
use  the  Defense  Acquisition  Guidebook  (Chapter  4)  as  a  framework  for 
improving  its  Systems  Engineering  guidance  because  it  is  complete  from  a 
process  viewpoint  and  is  supported  by  DoD  (the  fonner  USD/AT&L;  now  the 
SecAF). 

•  Very  few  AFRL  technology  program  leads  follow  the  AFRL  Instruction  or  a 
complete  set  of  Systems  Engineering  processes.  This  has  led  to  problems  in 
requirements  management,  risk  management,  and  other  areas  that  sometimes 
result  in  poor  program  performance  (including  schedule/cost  overruns)  and 
transition. 

•  Systems  Engineering  is  not  foreign  to  AFRL  personnel.  Although  it  is  not 
widespread  or  consistently  practiced,  institutionalizing  Systems  Engineering 
processes  should  not  be  as  difficult  as  if  they  were  concepts  new  to  AFRL. 


The  team  also  discovered  that: 
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•  Some  AFRL  personnel  are  concerned  that  Systems  Engineering  processes  are 
focused  on  acquisition  (as  opposed  to  research)  programs  and  might  stifle  the 
creative  atmosphere  essential  to  the  discovery  of  new  technologies. 

•  AFRL  has  a  requirement  to  implement  robust  Systems  Engineering  processes  in 
support  of  the  DoD  and  AF  acquisition  process.  DoD  has  recommended  a  series 
of  “best  practices”  for  Systems  Engineering  (described  in  Chapter  4  of  the 
Defense  Acquisition  Guidebook). 


•  Current  AFRL  Systems  Engineering  guidance  (AFRLI  61-104)  is  not  adequate  to 
ensure  such  a  Systems  Engineering  process.  In  addition  to  not  being  implemented 
by  most  programs,  it  does  not  address  a  sufficient  number  of  Systems  Engineering 
sub  processes. 

•  Most  Systems  Engineering  practices  are  represented  somewhere  amongst  the  set 
of  programs  and  ATDs  assessed,  so  the  core  understanding  of  good  Systems 
Engineering  practices  exist  today  in  pockets  throughout  AFRL. 

•  ATDs  and  other  programs  are  most  successful  when  they  have  both  strong  initial 
processes  (requirements  development  and  decision  analysis)  and  ongoing 
processes  to  address  requirements  changes  and  risk.  Integrated  Product  Teams 
(IPTs)  that  include  all  stakeholders  are  essential. 

•  Programs  have  the  most  difficulty  with  transitioning  technology  to  acquisition 
customers  and  warfighting  users.  This  is  due  in  part  to  changes  in  customer 
priorities  and  funding 

•  Uniqueness  in  the  way  AFRL  performs  S&T  programs  lies  not  in  what  they  are 
developing  or  how  they  develop  technologies;  rather  AFRL’s  unique  nature  lies  in 
how  it  focuses  its  energies  on  the  front  and  back  end  of  the  Systems  Engineering 
process,  making  much  of  the  intermediate  functions  the  responsibility  of  the 
contractor. 

•  The  Technology  Directorates  have  many  best  practices  that  can  be  used  by  the 
rest  of  AFRL. 

This  project  focused  on  S&T  projects  within  AFRL  however  all  of  these  findings  do 
apply  to  the  SBIR  community  within  AFRL.  They  identify  the  risk  of  inconsistent 
application  of  Systems  Engineering  and  failure  to  follow  best  practices.  The  findings 
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also  identify  that  current  SE  policy  is  not  adequate  to  ensure  good  SE  processes  are  being 
followed.  This  validates  further  the  need  to  develop  a  tailored  approach  for  the  SBIR 
community  since  SBIR  is  even  more  unique  then  typical  S&T  projects. 

Case  Study  Summary 

The  past  research  indentified  successfully  implemented  SE  processes  for  their 
projects.  The  studies  had  these  key  SE  attributes: 

•  Formed  a  multi-disciplinary  team  that  involved  all  relevant  stakeholders 

•  Held  team  reviews  to  monitor  project  progress 

•  Successfully  tailored  their  SE  approach  using  a  streamlined  S&T  process  tailored 
to  their  project  that  was  consistent  with  Air  Force  policy 

In  addition  the  TASE  report  validated  the  need  to  develop  a  tailored  SE  approach  for  the 
SBIR  community  as  it  identified  many  weaknesses  in  the  S&T  community  and  policy  for 
good  implementation  of  SE  processes.  It  highlighted  that  current  policy  is  not  sufficient 
with  AFRL  and  that  varying  levels  of  SE  are  being  implemented.  The  report  also  noted 
that  AFRL  relies  heavily  on  the  contractor  to  complete  many  SE  tasks  as  is  true  with 
SBIR  thus  making  it  unique.  Overall  the  literature  review  identified  many  pertinent  SE 
processes  for  the  SBIR  community  as  well  some  of  the  struggles  within  the  S&T 
community  to  fully  integrate  good  SE  processes. 
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III.  Methodology 


Chapter  Overview 

The  purpose  of  this  chapter  is  to  develop  the  methodology  for  analysis. 

Following  the  topic  selection  for  this  research  the  author  began  his  literature  review.  The 
first  step  was  to  review  all  past  research,  SBIR  documentation  and  applicable  SE  policy. 
Several  past  research  efforts  on  AFRL  projects  were  identified  to  be  relevant.  Figure  6 
illustrates  the  approach  that  was  developed  to  gather  and  analyze  data. 

Preliminary  review  of  AFRL  SBIR  projects  quickly  identified  varying  degrees  of 
SE  documentation  among  the  different  directorates.  In  most  cases  there  was  little  if  any 
SE  documentation.  Within  AFRL  projects  are  required  to  submit  a  Fonn  2913 
Laboratory  Management  Review  at  the  beginning,  annually  and  end  of  a  phase.  Some 
directorates  require  the  project  managers  to  answer  the  Eight  SE  Key  Questions  identified 
in  AFRL  policy  as  an  attachment.  A  color  scale  was  used  to  assess  each  question.  Out  of 
approximately  two  dozen  reviewed  all  of  them  showed  a  green  status  for  all  tasks.  No 
additional  comments  were  documented  for  any  of  them.  Project  Managers  were  only 
required  to  provide  comments  for  yellow  and  red  status  questions.  These  became  small 
vignettes  however  it  was  evident  that  additional  data  would  need  to  be  collected.  Other 
documentation  for  SBIR  includes  proposals,  contracts  and  final  reports.  SE  was  found  to 
not  be  well  documented  for  SBIR  projects  within  AFRL. 

The  lack  of  SBIR  SE  documentation  identified  that  additional  data  would  be 
required.  An  interview  instrument  to  collect  data  from  SBIR  project  managers  was 
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developed  and  participants  from  different  organizations  were  identified.  This  approach  is 
illustrated  below: 


Figure  6:  Grounded  Theory 

Once  all  data  had  been  collected  results  were  analyzed  to  identify  the  applicable 
SE  tasks  for  SBIR  projects.  This  research  is  an  exploratory  work  using  a  mixture  of 
qualitative  and  quantitative  methods  in  addition  to  using  existing  written  materials  as 
evidence.  With  the  help  of  the  SBIR  program  office  ideal  participants  managing  or 
overseeing  SBIR  projects  were  identified  for  the  interviews.  The  interview  is  semi- 
structured  so  that  open-ended  responses  were  encouraged  and  snowball  sampling  could 
occur.  Using  data  gathered  from  the  literature  review  and  interviews  enabled  a  structured 
content  analysis  through  triangulation  to  define  data  and  the  SE  rigor  that  is  associated 
with  SBIR  projects.  Using  a  triangulation  analysis  approach  enabled  the  author  to 
improve  the  validity  and  reliability  of  this  research  [Glafshani,  2003]. 
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Research  Objectives 

The  objective  of  this  research  project  is  to  define  the  SE  rigor  that  should  be  best 
applied  for  SBIR  projects.  To  define  the  rigor  and  design  a  tailored  approach  will  require 
identifying  what  degree  of  SE  is  applicable  in  the  SBIR  environment  and  how  those  SE 
processes  vary  amongst  projects  with  respect  to  project  maturity,  size  and  other  factors. 

Research  Questions 

To  define  the  research  objectives  the  following  question  must  be  answered: 

1 .  How  well  do  SBIR  projects  currently  implement  Systems  Engineering  policy? 

2.  How  are  SE  policies  implemented? 

3.  How  do  DoD  and  Air  Force  SE  processes  apply  to  SBIR  projects? 

4.  To  what  level  of  rigor  does  each  SE  process  apply  to  SBIR  projects? 

5.  What  is  the  best  way  to  implement  these  processes? 

Hypothesis 

Different  organizations  implement  varying  levels  of  Systems  Engineering  into 
their  SBIR  projects.  Organizational  SE  policies  and  SE  knowledge  base  vary  amongst 
SBIR  project  managers.  Therefore,  projects  are  at  risk  to  fail  meeting  DoD  and  AF 
standards  with  implementation  of  SE  and  SE  processes.  This  research  is  going  to  test  or 
measure  the  degree  to  which  SE  processes  are  applied  to  SBIR  projects.  Then  I  will 
analyze  the  results  to  determine  commonalities  and  differences  between  organizations. 
From  this  I  hope  to  generalize  working  SE  principles  for  the  SBIR  community. 
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Interview  Instrument  Development 

As  identified  in  the  literature  review,  AF  SEAM  fully  defines  the  SE  process 
involved  in  a  major  acquisition  program.  However  these  processes  are  defined 
differently  in  the  DAG  and  AFI  63-1201  as  illustrated  earlier  in  Table  1.  AFRL  policy 
maps  the  DAG  process  to  their  Eight  SE  Key  Questions.  The  interview  was  created  with 
this  understanding  and  is  designed  to  map  directly  back  to  AFRL  policy  that  maps  to  the 
DAG.  One  question  for  each  SE  process  outlined  in  the  DAG  was  created  with  the 
specific  tasks  called  out  in  AFRL  policy  identified.  The  interview  instrument  is  attached 
in  Appendix  1 .  Additional  questions  were  asked  about  the  effectiveness  of  the  Eight  SE 
Key  Questions  outlined  in  AFRL  policy.  AFRL  identifies  that  the  Eight  SE  Key 
Questions  guide  project  managers  in  implementing  the  SE  process. 

The  areas  targeted  during  the  interview  were: 

1 .  Job,  Organization,  Experience,  APDP  Education,  SBIR  Experience  for 
demographic  analysis 

2.  Stakeholders  Requirements  Definition 

3.  Requirements  Analysis 

4.  Requirements  Management 

5.  Decision  Analysis 

6.  Technical  Planning 

7.  Technical  Assessment 

8.  Technical  Data  Management 

9.  Risk  Management 

10.  Configuration  Management 

1 1 .  Interface  Management 

12.  Architectural  Design 

13.  Implementation 

14.  Integration 

15.  Verification 

16.  Validation 

17.  Transition 

18.  Usefulness  of  the  AFRL  Eight  SE  Key  Questions 
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Test  Subjects/Sample  Size 

The  day  to  day  management  of  SBIR  projects  for  the  government  is  typically 
performed  by  a  single  SBIR  project  manager.  These  project  managers  are  engineers  or 
program  managers  with  varying  levels  of  experience.  In  some  cases  the  project  manager 
may  have  little  experience  working  in  the  government  and  even  less  experience  with 
systems  engineering.  In  other  cases  they  have  been  working  in  the  government  for  30+ 
years.  Additionally  Chief  Engineers  for  the  directorates  oversee  the  management  of  these 
projects.  The  ideal  participants  for  the  interview  were  identified  as  SBIR  Project 
Managers,  Engineers  and  Chief  Engineers  because  they  are  the  most  familiar  with  the 
daily  technical  management  of  a  SBIR  project.  The  project  goal  was  to  interview  the 
Chief  Engineer  and  several  project  managers/engineers  from  each  organization. 
Participants  for  the  interview  were  identified  through  purposeful  and  snowball  sampling. 
Interviews  started  with  two  local  directorates  that  were  very  supportive  of  this  SE 
research  project  to  establish  a  baseline.  Additional  interview  participants  were  selected 
from  multiple  organizations  to  represent  the  broader  SBIR  community.  The  following 
AFRL  Technology  Directorates  participated:  Materials  Directorate,  Propulsion 
Directorate,  711  Human  Performance  Wing,  Space  Vehicles  Directorate  and  Munitions 
Directorate.  Organizations  outside  of  AFRL  that  manage  SBIR  projects  that  participated 
included:  Hill  AFB  Robins  AFB  Air  Logistics  Centers,  Arnold  AFB  Test  Center  and 
WPAFB  Aeronautical  Systems  Center. 
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Summary 

Analysis  of  the  interview  data  and  SE  artifacts  discovered  in  the  literature  review 
will  allow  identification  of  SE  processes  and  best  practices  within  the  SBIR  community 
using  a  triangulation  method  during  analysis.  Using  this  method  will  help  to  validate 
results  from  the  different  sources.  The  results  collected  from  the  interviews  will  also  help 
to  gauge  the  level  of  SE  rigor  being  implemented  within  the  different  organizations 
managing  SBIR  projects  and  help  to  identify  the  level  of  rigor  needed  for  applicable  SE 
tasks  for  SBIR  projects.  Comparing  those  results  with  current  guidance  and  policy 
discussed  in  Chapter  II  will  identify  the  current  Systems  Engineering  gap  in  policy  for  the 
SBIR  Community  and  allow  the  author  to  compare  and  contrast  the  current  policy  with 
the  research  findings  to  identify  and  develop  a  tailored  SE  approach  for  the  SBIR 
community  that  aligns  with  DoD  and  Air  Force  guidance  and  policy. 
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IV.  Analysis  and  Results 


Chapter  Overview 

This  chapter  captures  results  from  the  interviews  conducted.  It  provides  a 
consolidated  representation  of  results  as  well  a  detailed  analysis  for  each  SE  process  area. 
Using  results  from  the  interview  provides  a  SBIR  community  analysis  of  the  Eight  SE 
Key  Questions  and  it  answers  the  research  questions  outlined  in  Chapter  III. 


Interview  Results 

Interviews  conducted  varied  in  size  from  one  to  several  participants.  The  data 
was  collected  by  interviewing  each  organization  separately.  AFRL  XP  helped  to  identify 
potential  survey  participants  for  each  organization.  Participants  selected  were  SBIR 
Project  Engineers/Program  Managers  and  Chief  Engineers.  The  following  number  of 
participants  represented  each  organization  and  the  interview  results  color  scale  is  seen 
below: 


Table  6:  Total  Participants 


Technology  Directorates 

Test 

Centers 

Air 

Logistics 

Centers 

Other 

Materials  & 
Manufacturing 

Propulsion 

Space 

Vehicles 

Human 

Effectiveness 

Munitions 

Arnold 

Robins 

ASC 

4 

7 

3 

2 

2 

2 

3 

1 
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Table  7:  Interview  Results  Color  Scale 


Usually  not 
accomplished 


Sometimes 

accomplished 


Usually  accomplished 


Almost  always 
accomplished 


Usually  not  accomplished:  Less  than  25%  participants  identified  it  as  applicable 
Sometimes  accomplished:  25-50%  of  participants  identified  it  as  applicable 
Usually  accomplished:  50-75%  of  participants  identified  it  as  applicable 
Almost  always  accomplished:  75%  or  greater  of  participants  identified  it  as  applicable 

When  interpreting  the  below  results  make  sure  to  consider  that  the  specific  SE 
tasks  were  derived  from  AFRL  policy.  Air  Logistics  Centers  and  Test  Centers  have  a 
different  focus.  Thus  some  of  the  tasks  in  the  interview  are  in  S  &  T  tenns  which  may 
not  apply  in  some  areas  as  they  are  written  for  the  ALC’s  and  Test  Centers.  The 
additional  comments  documented  for  each  question  may  better  represent  the  SE  tasks 
currently  being  accomplished  in  those  cases. 
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Stakeholders  Requirement  Definition 

Overall  results  for  Stakeholders  Requirements  Definition  were  very  positive. 
Most  of  the  participants  identified  with  the  tasks  defined  in  AFRL  Policy.  ALCs 
however  did  not  identify  as  well  with  these  tasks. 


Table  8:  Stakeholders  Requirement  Definition 


Stakeholders  Requirement  Definition 

Technology 

Directorates 

Test 

Centers 

ALCs 

Total 

o  All  inputs  from  relevant  stakeholders 
translated  into  technical  requirements. 

71% 

100% 

33% 

70% 

o  Requirements  made  quantifiable,  have  unique 
definitions,  and  specified  thresholds  and 
objectives. 

71% 

100% 

67% 

75% 

o  Work  with  the  user  to  establish  and  refine 
goals,  attributes,  performance  parameters,  and 
financial  and  schedule  constraints,  and  then 
ensure  that  all  relevant  requirements  are 
addressed  during  the  science  and  technology 
effort. 

71% 

33% 

60% 

o  Translate  the  “customer  needs”  into  S&T 
program  and  system  requirements. 

79% 

100% 

33% 

75% 

Noteworthy  or  Significant  Comments 

Technology  Directorates  stated: 

•  Review  PEO/TEO  Needs  with  WBS  managers  and  Chief  Engineers. 

•  Evaluate  cost,  feasibility  (are  the  users  requirements  attainable  and  can  the 
company  actually  deliver  the  product  with  their  organic  capability?)  and  the 
technology  maturity  level. 

•  Work  to  refine  requirements  by  interacting  with  the  sponsor. 

•  Identify  likely  transition  and  what  requirements  are  needed  to  give  the  tech  a 
chance  at  transition 

Test  Centers  stated: 

•  We  attempt  to  include  other  government  agency  requirements  if  applicable  (ie 
Edwards  AFB  /  NASA  Aeronautics/  etc. 

•  In  general  we  do  all  of  these,  however  we  do  not  include  schedule  constraints  into 
our  planning.  Schedule  estimates  for  S&T  efforts  are  too  uncertain  to  include 
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them  in  our  planning.  Also,  most  of  these  are  really  only  applicable  to  the  Phase  II 
and  beyond  efforts.  Phase  I  is  to  show  us  your  capabilities,  whereas  Phase  II  is 
where  we  really  drive  technical  requirements. 

Air  Logistics  Centers: 

•  Develop/lead  technology  development/transition  teams  to  navigate  R&D  to 
implementation. 

•  Our  requirements  are  started  in  house  as  we  perform  the  maintenance  on  all  their 
assets.  Our  projects  are  based  on  reducing  total  ownership  costs  from  what  we 
see  in  the  field. 


Even  though  most  inputs  and  comments  identified  with  these  tasks  about  30%  of 
participants  did  not  identify  with  the  SE  tasks  identified.  Sponsor  involvement  and 
requirements  definition  to  ensure  valid  requirements  are  being  derived  to  meet  the 
operation  need  is  an  essential  part  of  acquiring  a  successful  system.  Thus  all  participants 
should  have  identified  with  these  tasks.  Better  education  of  SE  principals  would  likely 
help  community  to  better  identify  with  importance  of  these  tasks. 


Requirements  Analysis 

Technology  Directorates  identified  well  with  the  identified  tasks.  ALCs  and  Test 
Centers  did  not  relate  to  the  tasks  as  written  in  AFRL  language.  Also  the  scope  of  SBIR 
projects  can  vary  considerably  from  location  so  the  task  wording  my  not  seem  applicable 
as  phrased  in  some  cases. 
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Table  9:  Requirements  Analysis 


Requirements  Analysis 

Technology 

Directorates 

Test 

Centers 

ALCs 

Total 

o  Obtain  sets  of  logical  solutions  to  improve 
understanding  of  the  defined  requirements  and 
the  relationships  among  the  requirements  (e.g. 
functional,  behavioral,  temporal). 

71% 

33% 

55% 

o  Performance  parameters  and  constraints 
allocated  and  derived  technical  requirements 
defined. 

86% 

100% 

80% 

o  Partition  the  technical  problem  into  self- 
contained,  cohesive,  logical  groupings  of 
elements  and,  where  appropriate,  defined  the 
key  interfaces. 

50% 

33% 

40% 

Noteworthy  or  Significant  Comments 

Technology  Directorates  stated: 

•  Don’t  identify  a  solution,  just  help  with  providing  the  expertise. 

•  Sounds  a  bit  “ivory  tower”  (not  the  way  things  really  work).  We  look  at  each 
project  on  its  own  relative  merits.  We  have  to  make  an  inexact  mental  calculus  on 
the  impacts  a  successful  proposal  could  have  in  addressing  (1)  primary  sponsoring 
customers  needs,  (2)  broader  needs  of  USAF.  We  typically  work  with  these  same 
customers  throughout  the  program  and  try  to  maximize  relevance.  And  it’s  more 
complicated  than  that.  Sometimes  we  have  to  work  with  requirements  in  a  more 
“diffuse”  manner.  A  new  rad-hard  memory  chip  for  example,  may  not  have  a 
direct  connection  to  requirements  at  the  customer  level,  but  trickle  down  through 
specific  developments  involving  memory  chips  that  would  benefit.  We  would 
never  get  that  from  a  direct  customer.  This  is  the  essence  of  being  a  technical 
expert  in  a  laboratory  organization  and  working  across  a  longer  temporal 
perspective. 

•  Try  to  get  the  most  capability  that  can  reasonably  be  expected. 


Air  Logistics  Centers  stated: 

•  Map  the  R&D  into  self-contained,  cohesive,  logical  groupings  which  can  be 
incrementally  funded  seek  funding. 

•  Our  primary  focus  is  reducing  the  maintenance  costs,  total  ownership  costs. 
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Responses  for  these  SE  tasks  were  mixed.  These  tasks  were  proved  to  be  more 
applicable  to  the  Technology  Directorates.  Good  Requirements  Analysis  is  an  essential 
part  of  any  system  to  ensure  that  requirements  map  back  to  valid  performance  parameters 
and  constraints.  Results  identify  that  the  task  “Performance  parameters  and  constraints 
allocated  and  derived  technical  requirements  defined”  is  valid  for  SBIR  projects. 

Requirements  Management 

Overall  the  majority  of  participants  identified  with  the  listed  tasks  for 
Requirements  Management.  However  the  ALC’s  and  Test  Centers  did  not  in  all  cases. 
SBIR  phases  are  short  in  duration  and  requirements  typically  to  do  not  formally  change 
during  an  early  phase  which  likely  accounts  for  some  of  the  negative  responses. 

Table  10:  Requirements  Management 


Requirements  Management 

Technology 

Directorates 

Test 

Centers 

ALCs 

Total 

o  Maintain  the  traceability  of  all  requirements 
from  needs 

79% 

100% 

100% 

80% 

o  Document  all  changes  to  those  requirements 

64% 

33% 

50% 

o  Record  the  rationale  for  those  changes. 

64% 

33% 

50% 

o  Traceable  to  some  current  or  potential  future 
military  capability  need. 

71% 

100% 

33% 

70% 
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Noteworthy  or  Significant  Comments 

Technology  Directorates  stated: 

•  I  do  all  of  these,  but  do  not  document.  Just  keep  in  my  mind  and  notes  of  progress 
of  project  and  how  the  project  is  leaning  towards  meeting  an  application 
requirement,  etc. 

•  Sponsors  are  involved  with  requirement  changes. 

•  Increase  in  scope  done  through  interacting  with  sponsor  and  contractor. 

•  Letters  and  email  are  used  for  documenting  small  changes. 

•  Contract  changes  are  required  for  significant  deviation. 

•  It  is  still  an  inexact  calculus.  Customer  needs  are  not  necessary  sufficiently 
precise  for  the  exercises  you  believe  happen  in  requirements  management.  In  a 
Phase  1  for  example,  the  time  frame  is  almost  like  an  impulse  function  (a  single 
snapshot  in  time),  like  less  than  one  fiscal  year.  Only  Phase  2  projects  have  a 
gestation  interval  long  enough  to  matter  in  terms  of  evolving  needs.  It  is  a 
judgment  call  at  that  point,  and  we  find  some  Phase  2  projects  are  /  are  not 
flexible  enough  to  respond  to  changing  needs.  Using  significant  changes  are  not 
possible,  since  the  company  could  indicate  that  the  scope  changes  impact  ability 
to  deliver.  In  some  cases,  we  may  even  identify  alternate  /  additional  customers. 

Test  Centers  stated: 

•  The  end  user  makes  all  decisions  on  requirements.  In  general  we  require  the  end 
user  to  either  be  at  all  technical  reviews  or  be  the  project  manager. 


Inputs  and  comments  confirm  that  these  tasks  are  valid  in  the  SBIR  environment. 
SBIR  requirements  are  captured  at  a  top  level  and  tied  to  the  research  objectives  and  do 
not  change  typically  within  the  short  scope  of  the  phase.  Requirements  Management  is 
perfonned  with  increased  rigor  in  later  phases  of  the  project  as  it  becomes  more  relevant. 
However  Technology  Directorates,  Test  Centers  and  ALCs  should  be  accomplishing  all 
of  these  tasks  when  ever  requirements  change. 
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Decision  Analysis 

Results  from  the  interviews  identified  that  the  tasks  identified  for  Decision 
Analysis  are  not  always  accomplished.  Responses  were  higher  for  Technology 
Directorates  in  most  cases. 


Table  11:  Decision  Analysis 


Decision  Analysis 

Technology 

Directorates 

Test 

Centers 

ALCs 

Total 

o  Criteria  selected  for  decision  and  methods  to 
be  used  in  conducting  the  analysis. 

64% 

33% 

55% 

o  Analysis  conducted  to  help  choose  among 
alternatives  to  achieve  a  balanced,  supportable, 
robust,  and  cost  effective  program. 

64% 

33% 

50% 

o  Analysis  methods  include  some  of  the 
following:  trade  studies,  modeling  and 
simulation,  cost/benefit  analysis,  and  the 
analytic  hierarchy  process  (AHP). 

50% 

100% 

50% 

o  Studies  are  augmented  with  virtual  and/or 
physical  prototypes,  where  applicable,  prior  to 
making  decisions  on  best  alternative. 

50% 

67% 

45% 

Noteworthy  or  Significant  Comments 

Technology  Directorates  stated: 

•  I  look  at  the  performance  of  the  small  business,  the  likelihood  of  transition,  the 
approach,  and  for  innovation. 

•  Identify  the  best  approach  through  a  multi-disciplinary  team  Feasible  cost. 

•  Trade  studies  are  typically  not  accomplished. 

•  Like  to  use  modeling  and  simulation  when  applicable. 

•  This  “criteria”  business  does  not  track  with  SBIR  award  selection  criteria.  We 
are  bound  by  law  to  follow  the  fairly  vague,  broad,  and  PUBLISHED  criteria  (e.g. 
technical  merit,  experience,  dual-use/commercial/transition  potential. 

•  The  projects  themselves  may  embed  technical  trades,  modeling,  feasibility 
demonstration,  prototype  development. 


Responses  to  the  interview  and  additional  comments  varied  for  Decision 


Analysis.  A  limited  amount  of  Decision  Analysis  is  accomplished  during  Phase  I  and  II 
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SBIR  projects  due  to  the  limited  scope.  Focus  is  on  translating  the  research  into 
solutions.  SBIR  projects  must  work  within  the  limited  time  frame  and  resources 
allocated.  Formal  analysis  methods  and  prototypes  are  used  only  when  applicable  and 
the  project  has  the  resources. 


Technical  Planning 

Technical  Planning  establishes  and  maintains  documentation  that  defines  the 
technical  aspects  of  the  project.  Participants  identified  well  the  need  to  define  the  scope 
of  the  effort  to  include  exit  criteria,  constraints  and  interfaces. 


Table  12:  Technical  Planning 


Technical  Planning 

Technology 

Directorates 

Test 

Centers 

ALCs 

Total 

o  Technical  Planning  made  to  ensure  that  the 
technical  activities  were  conducted  properly 
throughout  the  system’s  life  cycle. 

43% 

30% 

o  Define  the  scope  of  the  technical  effort 
required  to  achieve  program  technical  goals 
which  includes  exit  criteria  and 
products/deliverables  which  can  be  tracked  with 
progress  measured. 

93% 

100% 

67% 

90% 

o  Indentify  constraints  and  interfaces  that  will 
result  in  derived  technical  requirements. 

64% 

100% 

67% 

70% 

o  Contribute  input  to  the  Systems  Engineering  1 

Plan,  which  is  owned  and  maintained  by  the  ]  C 

acquisition  activity. 

100% 

33% 

Noteworthy  or  Significant  Comments 

Technology  Directorates  stated: 

•  I  don’t  know  what  these  mean.  The  planning  goes  as  far  as  informing  the  SBIR 
awardees  the  expectations,  why  I  like  their  innovative  ideas,  and  any  adjustments 
I  think  should  be  taken  based  on  the  apparent  effectiveness  of  their  proposed 
work. 

•  Time  tables  are  associated  with  budget. 
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•  Reoccurring  interaction  with  contractor  is  needed. 

•  IMP  does  not  apply  for  projects  under  20M. 

•  Prior  to  proposal  following  topic  call  RZ  provides  feedback  on  expected 
evaluation  criteria. 

•  I  am  not  even  sure  what  this  means.  We  typically  negotiate  topics  with  sponsors, 
along  the  lines  of  requirements.  These  topics  usually  (but  not  always)  track  with 
their  top  priorities.  It  is  subjective,  because  we  and  they  are  human.  Sometimes 
as  a  result  we  live  with  “not  the  best  defined”  topics,  and  we  have  to  do  the  best 
we  can  to  ensure  that  the  work  is  relevant.  On  the  scale  of  the  SBIR  program,  it  is 
impossible  to  get  this  right  100%  of  the  time. 

Test  Centers  stated: 

•  For  AEDC,  the  end  user  is  also  the  acquisition  activity.  Our  SBIR’s  are  designed 
with  a  specific  AEDC  use  in  mind.  We  try  to  incorporate  other  centers’  technical 
requirements  to  enable  better  commercialization  but  these  are  secondary  to  the 
AEDC  requirements. 

Technical  Planning  for  throughout  the  life  cycle  and  development  of  a  Systems 
Engineering  Plan  (SEP)  are  tasks  more  applicable  later  in  the  development.  Results  for 
those  areas  identified  them  to  not  be  applicable  for  SBIR  phase  I  and  II  projects.  Though 
a  particular  SEP  may  not  exist  for  those  phases’  project  managers  should  identify  and 
follow  the  overarching  SEP  if  one  exists  for  the  platfonn  that  the  SBIR  project  will 
eventually  integrate  to  or  the  overarching  organizational  SEP. 


Technical  Assessment 

Several  of  the  tasks  for  Technical  Assessment  were  identified  during  the 
interviews  as  displayed  in  green  below.  However  two  of  the  tasks  (highlighted  in  yellow) 
were  identified  as  not  always  applicable. 
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Table  13:  Technical  Assessment 


Technical  Assessment 

Technology 

Directorates 

Test 

Centers 

ALCs 

Total 

o  Measure  technical  progress,  technology 
maturity,  and  the  effectiveness  of  plans  and 
requirements.  Activities  include:  Technical 
Performance  Measurement,  Technology 
Readiness  Assessment,  and  the  conduct  of 
technical  reviews. 

86% 

100% 

100% 

85% 

o  Demonstrate  and  confirm  completion  of 
required  accomplishments  and  S&T  exit  criteria. 

79% 

100% 

33% 

70% 

o  Discover  deficiencies  or  anomalies  that  may 
result  in  the  application  of  corrective  action  and 
may  have  formed  the  technical  portion  of  a 
continuous  process  improvement  process  when 
used  to  evaluate  application  of  SE  1AW 
paragraph  2. 5. 5.2. 

43% 

100% 

33% 

45% 

o  Technical  assessment  inputs  used  in  support 
of  the  Laboratory  Management  Review  process. 

86% 

71% 

o  Technical  assessment  activities  conducted  in 
concert  with  existing  reviews  where  possible  to 
minimize  disruption  to  the  research  project. 

36% 

33% 

35% 

Noteworthy  or  Significant  Comments 

Technology  Directorates  stated: 

•  As  the  contract  program  manager,  I  do  all  of  these  on  a  subjective  basis  and  a  gut 
feeling  using  engineering  judgment  of  best  approach. 

•  These  things  sound  like  they  come  from  textbooks  or  acquisition  training 
programs.  Most  SBIRs  are  so  short  in  duration  and  fluid,  not  to  mention  limited 
in  scope,  that  some  of  these  activities  (TRA)  are  too  difficult  to  do  on  a  recurring 
basis  (like  every  3-6  months).  Also,  it  is  not  uncommon  for  some  individuals  to 
have  more  than  12  SBIRs  at  any  moment.  They  are  usually  a  secondary  duty,  as 
we  cannot  afford  to  dedicate  even  a  single  individual  to  1  or  2  SBIRs. 

Test  Centers  stated: 

•  Every  AEDC  SBIR  is  required  to  have  an  onsite  demonstration  at  the  midpoint  of 
the  Phase  II.  We  also  encourage  our  contractors  to  provide  a  demonstration  at  the 
end  of  Phase  I  in  order  to  show  a  viable  path  to  the  required  TRL. 
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The  interview  results  and  comments  identified  the  applicable  tasks  during  phase  I 
and  II  SBIR  projects.  The  two  tasks  that  were  identified  in  yellow  may  be  applicable  in 
some  cases.  However  due  to  the  short  duration  and  limited  scope  of  the  SBIR  phase  they 
may  not  be  applicable.  Overall  Technical  Assessment  is  a  critical  step  in  the  technical 
management  of  the  project  to  ensure  the  project  is  mature  enough  to  enter  the  next  phase 
by  meeting  entry  and  exit  criteria  and  should  be  assessed  by  all  SBIR  project  managers. 


Technical  Data  Management 


The  SBIR  community  identified  the  formal  use  of  the  Defense  Technical 
Information  Center.  All  formal  reports  are  documented  in  DTIC. 


Table  14:  Technical  Data  Management 


Technical  Data  Management 


Technology 

Directorates 


Test 

Centers 


ALCs 


Total 


o  Project  data  managed  through  the  Defense 
Technical  Information  Center  (DTIC) 


100% 


100% 


33% 


85% 


Or  similar  data  base. 


Noteworthy  or  Significant  Comments 

Technology  Directorates  stated: 

•  We  have  a  fairly  regular  set  of  actions  that  are  always  performed.  We  have 
kickoff  meeting,  periodic  technical  interchange  meetings  and  telecons,  emails  on 
demand  for  technical  clarification,  we  review  these  in  LMRs,  we  document 
research  summaries  and  final  technical  reports  through  DTIC. 

•  We  also  respond,  on  customer  demand,  to  participate  in  industry  days  or  work 
with  their  own  database  initiatives,  which  come  and  go  over  time. 

Air  Logistics  Centers  stated: 

•  Final  reports  are  put  in  DTIC,  we  use  our  local  server  to  store  all  contract 
activities. 
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Risk  Management 

A  good  risk  management  strategy  is  critical  for  a  successful  project.  The  large 
majority  of  participants  identified  that  the  five  risk  management  steps  apply  to  the  SBIR 
community.  65%  identified  acknowledged  the  existence  of  a  risk  management  plan  for 
their  project.  However  feedback  from  the  interviews  identified  that  the  Small  Business 
perfonn  most  aspects  of  the  risk  management. 


Table  15:  Risk  Management 


Risk  Management 

Technology 

Directorates 

Test 

Centers 

ALCs 

Total 

o  Develop  risk  management  plan  and  performed 
the  following: 

57% 

100% 

100% 

65% 

o  Identified  risk 

86% 

100% 

100% 

85% 

o  Analyze  risk  and  define  probably  and  likelihood. 

64% 

100% 

100% 

70% 

o  Identify  handling  options 

64% 

100% 

100% 

70% 

o  Mitigate  risk 

79% 

100% 

100% 

80% 

o  Track  risk 

86% 

100% 

100% 

85% 

Noteworthy  or  Significant  Comments 

Technology  Directorates  stated: 

•  Risk  is  tracked  by  the  project  officer. 

•  Information  is  provided  by  contractor  (varies  on  how  often  and  well  PO  interacts 
and  follows  the  contractors  progress). 

•  AFP  AM  63-128  is  not  really  being  used  as  a  guide  formally.  Limited  knowledge 
of  it  in  work  environment. 

•  In  phase  1  s,  there  is  very  little  time  to  do  any  of  this  except  in  the  brief 
interchanges  that  one  can  have  over  a  6-9month  technical  activity.  They  either 
“get  it”  or  they  don’t,  in  which  case  you  request  /  don’t  request  a  Phase  2 
proposal.  This  is  Darwinian.  You  then  hope  that  the  best  ones  will  be  awarded, 
but  it  is  a  competitive  process,  so  sometimes  even  a  perfect  Phase  1  ends  without 
a  follow-on.  The  steps  you  outline  above  are  typically  done  in  a  very  “seat  of 
pants”  way,  but  they  typically  ARE  done. 

•  Risk  in  a  SBIR  is  generally  limited  to  technical  risk  as  cost  is  fixed  and  schedule 
deliverables  are  detailed. 
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Test  Centers  stated: 

•  Occasionally  use  canned  Risk  Analysis  software  (ie  Crystal  Ball). 

•  We  understand  every  SBIR  carries  programmatic  risk.  Therefore  these  activities 
are  only  done  for  Phase  Ill’s. 

•  For  SBIR,  we  mainly  use  canned  risk  analysis  software  and  occasionally  use 
Monte-Carlo  methods  to  conduct  risk  trade  analyses. 


Air  Logistics  Centers  stated: 

•  SBIR  programs  are  inherently  at  a  high  technical  risk  due  to  the  nature  of  new 
technological  development.  However,  risk  is  managed  by  the  outline  above.  I  am 
always  trying  to  identify  schedule,  cost  and  technical  risk  early  and  working  with 
the  contractor  to  mitigate.  Early  detection  is  essential  to  risk  management  and 
overall  program  success. 

•  Risks  on  our  projects  are  what  we  over  come  in  the  SBIRS  the  final  analysis  and 
subsequence  prototype  testing  will  validate  success. 


Responses  and  comments  identified  that  the  SBIR  community  is  aware  of  risk 
management  techniques  and  they  are  applicable  to  SBIR  projects.  Project  managers  in 
many  cases  rely  on  information  from  the  contractor  for  their  assessments.  Project 
managers  must  ensure  they  follow  up  on  a  regular  basis  with  the  contractor  to  check  the 
health  of  the  project  and  ensure  the  risk  is  being  managed  effectively  and  the  project  is  on 
scope. 


Configuration  Management 


Most  participants  identified  the  need  to  document  results  for  Configuration 
Management.  However  there  were  mixed  results  from  the  interview  responses. 
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Table  16:  Configuration  Management 


Configuration  Management 

Technology 

Directorates 

Test 

Centers 

ALCs 

Total 

o  Ensure  the  repeatability  of  experimental 
results  to  include  data  by  knowing  and  keeping  a 
record  of  the  laboratory  set-up  as  well  as 
tracking  changes  to  it. 

64% 

50% 

33% 

55% 

o  Keep  a  record  of  laboratory  experimental 
hardware  configuration  when  measurements  are 
gathered  including  such  things  as  calibration 
status,  environmental  conditions,  software 
version  and  modifications  used,  and 
documentation  (data)  resulting  from  the 
experiment/ demonstration? 

43% 

50% 

33% 

40% 

o  A  complete  audit  trail  of  decisions  affecting 
laboratory  equipment/software  design 
modifications  maintained. 

29%  0%  0%  20% 

Noteworthy  or  Significant  Comments 

Technology  Directorates  stated: 

•  None,  these  do  not  seem  to  apply  to  the  type  of  SBIR’s  I  manage.  Seems  more  in- 
house  research  as  opposed  to  SBIR  contracts  in  which  are  perfonned  by  external 
to  WPAFB  contractors. 

•  Contractor  maintains  CM  records.  PO  does  not  have  a  CMP. 

•  Gov’t  PO  is  not  very  involved  with  ensuring  CM  is  accomplished  on  average. 

•  Final  technical  report  is  published,  to  document  permanently  what  we  did  do  in 
terms  of  the  things  you  describe  above. 

•  Configurations  for  official  tests  and  demos  are  documented,  we  also  rely  on 
contractor  supplied  ICDs,  schematics,  etc. 

Test  Centers  stated: 

•  AEDC  only  configures  once  the  SBIR  is  ready  for  transition  to  an  actual  test  cell. 
This  normally  happens  at  least  one  year  following  the  close-out  of  the  phase  II. 
For  Phase  III  acquisitions,  AEDC  follows  regular  SE  processes  and  requires  full 
configuration  management  as  part  of  the  acquisition. 

Air  Logistics  Centers  stated: 

•  Perfonned  by  the  SBIR  contractor,  the  information  is  contained  in  their  interim 
reports,  also  much  of  the  testing  is  perfonned  by  an  independent  testing  facility, 
which  will  identify  the  equipment  used  and  calibrations.  These  data  sheets  are 
included  with  the  reports. 
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Results  and  comments  are  concerning  since  it  seems  that  Configuration 
Management  is  dependent  on  the  contractor  and  not  well  regulated  by  the  SBIR  project 
managers.  A  fundamental  basic  of  good  SE  is  to  maintain  good  configuration 
management  for  the  life  cycle  of  the  system.  S&T  projects  are  derived  to  support  a 
potential  capability  need  or  requirement.  Thus  all  project  results  should  be  documented 
to  ensure  traceability  to  the  higher  level  requirements  as  well  repeatability  of  results. 
Lack  of  proper  documentation  and  traceability  risks  wasting  efforts  that  do  not  support 
the  project  objectives  as  well  as  increase  the  challenges  of  transitioning  to  the  next  phase 
with  well  documented  repeatable  results. 


Interface  Management 

Similarly  to  Configuration  Management  responses  varied  and  not  many  Interface 
Management  tasks  were  not  validated  as  being  applicable. 


Table  17:  Interface  Management 


Interface  Management 


Technology 

Directorates 


o  Ensure  interface  definition  and  compliance 
among  the  elements  that  compose  the  laboratory 
system  (internal  interfaces),  as  well  as  with  other 
systems  with  which  the  operational  system  or 
system  elements  might  interact  (external 
interfaces). 


o  Ensure  that  all  internal  and  external  interface 
requirement  changes  are  properly  documented  in 
accordance  with  the  configuration  management 
plan  and  communicated  to  all  affected  elements 
of  the  program. 


o  All  interfaces  defined  in  sufficient  detail  to 
facilitate  necessary  communication/interaction 
among  system,  subsystem,  and  components. 


50% 


50% 


71% 
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Noteworthy  or  Significant  Comments 

Technology  Directorates  stated: 

•  None  apply  to  the  type  of  SBIR’s  I  manage.  My  SBIR’s  relate  to  component 
maturation  rather  than  system  engineering.  We  are  trying  to  demonstrate  a  higher 
perfonnance  component  where  we  need  to  be  aware  of  the  mating  technologies. 
However,  interfacing  usually  comes  much  beyond  Phase  II  and  often  times 
beyond  Phase  III  contracts. 

•  Defined  for  Demonstration  configuration  in  showing  program  met  the  topic  goals. 

•  Contractor  maintains  IM  records. 

•  There  should  be  stake  holder  involvement  to  ensure  interface  interactions  are 
defined  and  acceptable. 

•  Nothing  this  formal,  except  in  rare  occasions. 

Test  Centers  stated: 

•  AEDC  does  not  have  SBIRs  that  are  intended  to  be  inserted  into  other  systems 
and  therefore  do  not  require  ICD’s. 

•  When  we  transition  a  SBIR  all  of  the  above  occurs  but  not  during  a  SBIR. 

Air  Logistics  Centers  stated: 

•  We  work  with  structural  type  systems  (Shelters,  radomes  and  towers).  Each  is 
unique  to  the  individual  mission.  We  don’t  have  formal  external  interfaces.  We 
do  have  industry  standards  and  local  codes  for  geographic  location  that  they  must 
meet  and  pass. 


A  low  level  of  rigor  for  Interface  Management  may  only  be  required  for  many 
typical  SBIR  projects  because  not  all  interfaces  may  yet  be  fully  defined.  However 
stakeholder  involvement  and  early  architecture  efforts  should  identify  the  interfaces 
necessary  and  drive  the  requirements  for  early  Interface  Management.  Overall  the  SBIR 
community  did  not  identify  well  with  the  tasks  identified.  Evidence  suggest  that  SBIR 
project  managers  really  heavily  on  the  Small  Business  to  manage  interfaces.  This  is  a 
potential  risk  since  good  interface  management  is  required  to  be  successful  in 
transitioning  the  project. 
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Architectural  Design 


Responses  for  Architectural  Design  did  not  validate  any  of  the  tasks. 


Table  18:  Architectural  Design 


Technology  Directorates 


o  Translate  the  outputs  of  the  Stakeholder 
Requirements  Definition  and  Requirements 
Analysis  processes  into  alternative  technical 
solutions  and  selects  a  final  technical  path  to 
explore. 


Technology 

Directorates 


o  Iterate  Stakeholder  Requirements  Definition, 
Requirements  Analysis,  and  with  the  technical 
management  processes  to  identify  and  select  the 
best  solution  by  first  developing  a  high-level 
view  of  the  system  architecture  capable  of 
meeting  stakeholder  needs. 


o  Output  the  design  functional  or  physical 
architecture  sufficiently  detailed  to  allow 
upward  and  downward  traceability  of 
requirements. 


Noteworthy  or  Significant  Comments 

Technology  Directorates  stated: 

•  Architectural  design  is  not  accounted.  Rather,  we  design  to  meet  an  objective  of  a 
component  performance. 

•  Don’t  feel  architectures  apply  much  for  phase  I  and  II  projects. 

•  Architectural  design  is  not  accounted.  Rather,  we  design  to  meet  an  objective  of  a 
component  performance. 

•  Fonnal  Architectural  Design  rarely  has  use  in  the  world  of  SBIR  because  the 
customers  aren’t  concerned  about  this  sort  of  formal  representation  of  impact 
from  a  SBIR  effort.  I  am  sure  we  would  do  it  if  we  sensed  any  utility. 

Air  Logistics  Centers  stated: 

•  In  our  case,  the  stakeholder  comes  into  play  after  we  ensure  the  product  meets  the 
basic  standards  such  as  ANSI  1925  for  shelters.  Then  the  stakeholders  come  into 
play  to  incorporate  their  system  into  our  product. 


52 


Responses  and  comments  for  Architectural  Design  are  very  concerning.  S&T 
projects  risk  being  out  of  scope  with  DoD  objectives  without  maintaining  traceability  to 
requirements.  Early  architectural  mapping  to  those  requirements  ensure  that  projects 
stay  on  scope.  Results  identified  that  the  majority  of  participants  do  not  formally 
document  architecture  and  many  do  not  feel  architecture  is  applicable  to  SBIR  which  is  a 
discouraging  misconception.  Even  Basic  Research  should  map  to  a  high  level  capability. 


Implementation 

Capturing  the  right  information  to  address  the  preparation  required  to  support  the 
project  transition  to  the  next  phase  on  aspects  of  production  of  products  and/or  services  is 
an  important  part  of  the  project.  The  majority  of  participants  identified  with  the  below 
tasks.  However,  as  results  showed  below,  all  are  applicable  tasks  in  the  SBIR  community 
but  only  50  and  55%  identified  with  two  of  the  task. 


Table  19:  Implementation 


Implementation 

Technology 

Directorates 

Test 

Centers 

ALCs 

Total 

o  Yield  the  fundamental  capability  of  the 
program. 

64% 

50% 

33% 

55% 

o  Include  some  testing  of  the  individual 
elements  before  they  passed  to  Integration. 

79% 

100% 

67% 

75% 

o  Develop  supporting  documentation  for  the 
system;  such  as  the  as-built  configuration,  or 
discovered  limitations  of  the  concept;  is  also  a 
part  of  the  implementation  process. 

43% 

100% 

67% 

50% 
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Noteworthy  or  Significant  Comments 

Technology  Directorates  stated: 

•  We  ensure  we  are  meeting  some  sort  of  perfonnance  requirement  before 
attempting  to  look  at  integration  or  mating  with  other  technologies.  However,  this 
usually  comes  at  the  end  of  a  SBIR  program  where  there  are  not  usually  sufficient 
funds  left  for  maturing  any  more. 

•  Include  some  testing  of  the  individual  elements  before  they  passed  to  integration 

•  We  test  specific  assertions  that  comprise  the  program  statement  of  work.  To  the 
degree  they  embody  these  things,  we  do  them.  They  are  certainly  documented  - 
variously  -  through  the  status  reports  and  final  technical  reports  that  are  always 
required. 

Test  Centers  stated: 

•  AEDC  will  normally  accept  the  deliverable  and  test  it  in  a  lab  quite  some  time 
prior  to  transition  to  an  actual  test  cell. 

Air  Logistics  Centers  stated: 

•  Update  T.O.s  &  Drawings,  publish  new  ASTM,  etc.,  Standards. 

•  For  Joint-Service  Teams  to  implement  to  wider  DoD  activities. 

•  Implementation  is  accomplished  through  capturing  all  technical  requirements 
within  an  Air  Force  specification  which  then  rolls  up  into  Technical  Manuals 
(T.O.’s). 


Results  and  comments  overall  were  positive  for  Implementation.  Most  of  the 
SBIR  community  interviewed  understood  the  importance  of  Implementation.  It  however 
accomplished  with  limited  rigor  during  SBIR  phase  I  and  II  due  to  the  limited  scope  and 
early  development  effort.  Implementation  should  increase  in  rigor  as  the  project  matures. 


Integration 

Results  for  the  Integration  tasks  identified  that  the  tasks  are  typically  not 
applicable  in  the  SBIR  phase  I  and  II  environment. 
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Table  20:  Integration 


Integration 

Technology 

Directorates 

Test 

Centers 

ALCs 

Total 

o  Incorporate  the  lower-level  system  elements 
into  a  higher-level  system  element  in  the 
physical  architecture. 

36% 

50% 

100% 

45% 

o  Define  the  plan  or  strategy  for  the  Integration 
process,  including  the  assembly  sequence,  that 
may  have  imposed  constraints  on  the  design 
solution 

43% 

50% 

67% 

45% 

Noteworthy  or  Significant  Comments 

Technology  Directorates  stated: 

•  We  work  with  the  prime  contractors  to  see  if  there  is  interest.  If  there  is,  we  look 
for  funding  to  integrate  the  technology  into  some  relevant  system.  Unfortunately, 
things  usually  die  early  due  to  insufficient  funds. 

•  Integration  as  need  to  demonstrate  the  SBIR  goals  -  not  a  formal  plan  but  may  be 
art  of  the  test  plan. 

•  Do  not  dictate  the  process,  only  indicate  the  requirements. 

•  Work  with  the  contactor  to  see  if  there  is  interest  to  integrate  a  SBIR  into  a 
relevant  system  and  try  to  locate  funding. 

•  Very  limited  in  Phase  I  and  II  SBIRs. 

Air  Logistics  Centers  stated: 

•  Generally  we  take  other  “Systems”  and  incorporate  them  in  our  shelter,  put  on  a 
tower  or  cover  them  up  with  a  radome. 


As  results  and  comments  identified  a  larger  emphasis  on  integration  takes  place  in 
Phase  III  of  a  SBIR  project.  Thus  results  for  the  tasks  identified  from  AFRL  policy  for 
integration  were  not  identified  as  applicable  by  the  majority  of  participants  for  the  Phase  I 
&  II  SBIR  community.  These  tasks  may  or  may  not  be  applicable  depending  on  the 
maturity  of  the  project.  ALCs  did  identify  with  these  tasks  likely  because  they  have  more 
initial  infonnation  on  the  platform  the  SBIR  project  will  be  integrated  to  support. 
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Verification 


Responses  identified  that  SBIR  projects  verify  some  elements  of  the  project. 
However,  the  tasks  in  yellow  were  identified  as  not  applicable  for  SBIR  phase  I  and  II 
projects. 


Table  21:  Verification 


Verification 

Technology 

Directorates 

Test 

Centers 

ALCs 

Total 

o  Confirm  that  the  laboratory/experimental 
system  element  meets  design  specifications. 

86% 

50% 

100% 

80% 

o  Test  the  system  elements  against  their  defined 
requirements  (predicted  versus  experimental 
results). 

64% 

50% 

100% 

65% 

o  Design  solutions  at  all  levels  of  the  physical 
architecture  were  verified  through  a  cost- 
effective  combination  of  analysis,  examination, 
demonstration,  and  testing,  all  of  which  can  be 
aided  by  modeling  and  simulation. 

29% 

67% 

30% 

o  Answer  the  verification  question  “Did  we 
build  the  thing  right?”. 

29% 

50% 

67% 

40% 

Noteworthy  or  Significant  Comments 

Technology  Directorates  stated: 

•  Some  verification  takes  place  with  component  testing. 

•  Demos  are  usually  joint  test  events  and  we  try  to  test  the  capability  against 
operational  expectations. 

Air  Logistics  Centers  stated: 

•  Perfonn  “Live-Fire”,  and  shop  level  performance  and  testing  (Zinc -Nickel  plating 
to  replace  Cadmium  on  Landing  Gear  components). 

•  It’s  more  subtle  -  and  complex  -  than  these  simple  menu  choices. 

Results  and  comments  identified  that  SBIR  phase  I  and  II  projects  only 
accomplish  a  limited  amount  of  Verification  testing.  Further  verification  testing  will  be 

accomplished  in  later  phases.  Thus  answering  the  verification  question  “Did  we  build  the 

56 


thing  right?”  will  be  accomplished  later  in  the  development  of  the  project.  Similarly 
design  solutions  for  all  levels  of  the  physical  architecture  are  more  applicable  in  the  later 
phases  of  the  project. 


Validation 

Responses  identified  Validation  tasks  are  accomplished  for  SBIR  projects. 
However  the  validation  question  “Did  we  build  the  right  thing  was  identified  as  not 
always  applicable. 


Table  22:  Validation 


Validation 

Technology 

Directorates 

Test 

Centers 

ALCs 

Total 

o  Test  the  performance  of  the  technology 
against  the  original  program  goals. 

86% 

100% 

67% 

80% 

o  Capture  any  testing  results/data  so  that  they 
are  available  for  further 
development/research/maturation  efforts. 

79% 

100% 

67% 

75% 

o  Answer  the  validation  question  “Did  we  build 
the  right  thing?”. 

29% 

50% 

100% 

45% 

Noteworthy  or  Significant  Comments 

Test  Centers  stated: 

•  It  is  rare  in  a  SBIR  development  that  we  receive  exactly  what  we  want.  Clearly, 
the  small  business  is  interested  in  commercialization  rather  than  simply  delivery 
of  the  prototype.  Because  of  this,  AEDC  must  often  spend  additional  mission 
resources  to  ensure  the  system  meets  our  requirements.  We  do  this  through 
additional  development  once  the  prototype  has  been  received. 

•  Usually  not  enough  funding  through  a  SBIR  for  validation.  Full  validation  takes 
place  as  a  SBIR  is  integrated  into  a  system  through  other  funding  methods. 

•  Sometimes,  on  the  scale  of  a  SBIR,  we  don’t  even  get  a  WHOLE  thing,  and  we 
sometimes  cannot  answer  these  simplistic  questions  like  “did  we  build  the  right 
thing” 

•  All  are  done  informally. 
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SBIR  Phase  I  and  II  projects  only  accomplish  a  limited  amount  of  Validation 
testing  due  to  their  limited  scope  and  duration.  Further  validation  testing  will  be 
accomplished  in  later  phases. 


Transition 

Most  participants  identified  the  importance  of  considering  and  applying  steps  for 
their  projects  to  transition  to  the  next  phase. 


Table  23:  Transition 


Transition 

Technology 

Directorates 

Test 

Centers 

ALCs 

Total 

o  Deliver  a  supportable  technology  project 
capable  of  being  put  in  the  hands  of  the 
warfighter. 

43% 

50% 

67% 

45% 

o  The  transition  process  applied  in  a  step-by- 
step  manner  to  move  the  technology  to  the  next 
level  in  the  developmental  cycle. 

86% 

50% 

67% 

75% 

o  Needs  of  follow-on  phases  considered  early 
in  the  program  and  included  in  all  of  the 
technical  management  processes. 

71% 

50% 

100% 

75% 

Noteworthy  or  Significant  Comments 

Technology  Directorates  stated: 

•  Not  much  consideration  during  phase  I  and  phase  II. 

•  There  is  no  such  thing  as  a  “step  by  step”  transition  process.  This  seems  to  reflect 
a  fairly  meager  understanding  of  technology  development.  You  are  not  always 
able  to  mature  even  a  piece  of  a  problem  to  a  level  that  can  be  transitioned. 

•  Usually  you  target  a  large  defense  contractor/SPO/gov  agency  that  can  integrate 
the  tech  into  their  products  rather  than  straight  to  warfighter/production. 

Air  Logistics  Centers  stated: 

•  In  phase  II  we  focus  on  one  system  or  customer  that  has  a  need  for  the 
technology.  We  find  that  new  technology  is  met  with  resistance.  That  it  is 
important  to  have  a  working  prototype  to  customers  can  “kick  the  tires”  or  see  a 
physical  item. 
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As  results  and  comments  validated  Transition  tasks  for  SBIR  not  all  tasks  were 


identified  as  applicable.  “Deliver  a  supportable  technology  project  capable  of  being  put 
in  the  hands  of  the  warfighter”  is  an  action  that  takes  place  in  Phase  III  and  is  not  directly 
applicable  to  phase  I  &  II  projects  until  they  progress  to  that  phase.  That  is  why  only  45% 
of  the  participants  identified  with  that  task.  Some  SBIR  projects  never  reach  phase  III. 


Summary  of  Results 

Results  below  include  all  the  participants.  It  illustrates  that  how  SBIR  is  unique 
since  for  a  major  acquisition  program  all  tasks  would  be  applicable. 


%  of  Interview  SE  Tasks  Accomplished  by  all 
participants 
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Figure  7:  All  Participant  Results 
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Center  Results 


Results  identified  several  weak  areas  with  regards  of  implementation  of  AFRL  SE 
tasks  from  identified  in  AFRL  policy.  This  illustrates  how  SBIR  is  unique  when 
compared  to  typical  S  &  T  projects  within  AFRL. 


%  of  Interview  SE  Tasks  Accomplished  by  Technology 

Directorates 
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Figure  8:  Technology  Directorates  Results 
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Test  Center  Results 


Since  the  interview  instrument  was  developed  using  AFRL  policy  some  of  the 
tasks  were  in  AFRL  language  and  participants  did  not  identify  with  them.  Data  was 
gathered  from  a  small  participant  size  for  Test  Centers.  Both  of  those  factors  must  be 
considered  when  intrepreting  results.  Additional  comments  on  how  tasks  were  actually 
perfonned  was  captured  in  their  comments  that  was  previously  noted  for  each  area. 
Architectural  Design,  Decision  Analysis  and  Interface  Management  identified  no 
interview  SE  tasks  as  applicable  from  survey  responses. 


%  of  Interview  SE  Tasks  Accomplished  by  Test  Centers 
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Figure  9:  Test  Centers  Results 
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Air  Logistics  Center  Results 

Since  the  interview  instrument  was  developed  using  AFRL  policy  some  of  the 
tasks  were  in  AFRL  language  and  participants  did  not  identify  with  them.  Data  was 
gathered  from  a  small  participant  size  for  ALC’s.  Both  of  those  factors  must  be 
considered  when  intrepreting  results.  Additional  comments  on  how  tasks  were  actually 
perfonned  was  captured  in  their  comments  that  was  previously  noted  for  each  area. 


Figure  10:  Air  Logistics  Centers  Results 
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AFRL  Eight  Systems  Engineering  Key  Questions 

AFRL  emphasizes  the  use  of  the  Eight  SE  Key  Questions  identified  in  AFRLI  61- 
104.  The  questions  are  mapped  back  to  the  DAG  SE  processes.  This  below  table 
illustrates  the  interview  results  mapped  back  to  the  questions.  The  majority  of  interview 
participants  identified  these  questions  to  be  useful  to  extremely  useful.  Each  participant 
subjectively  identified  the  most  and  least  useful  question  illustrated  in  the  below  table. 


Table  24:  AFRL  SE  Key  Questions  Mapped  to  Interview  Results 


Most 

Least 

useful 

Useful 

Mapped  to  SE  Process 

1.  Who  is  your  customer? 

5 

2 

Requirements  Management 

Requirements  Definition 

Requirements  Management 

2.  What  are  the  customer’s 
requirements? 

5 

Configuration,  Data  &  Interface  Management 
Requirements  Development  &  Validation 

Technical  Planning  &  Technical  Assessment 
Decision  Analysis 

Risk  &  Requirements  Management 

Verification,  Validation  &  Transition 

3.  How  will  you  demonstrate  you 

Configuration  &  Interface  Management 
Integration 

have  met  the  requirements? 

5 

1 

4.  What  are  the  technology 

Technical  Planning  &  Implementation 

options? 

1 

2 

Technical  Assessment 

Decision  Analysis  &  Implementation 

Integration 

5.  Which  is  the  best  approach? 

1 

6.  What  are  the  risks  to 

Technical  Planning 

developing  the  selected 

Risk  Management 

technology? 

2 

1 

Risk  Management  &  Data  Management 

Technical  Planning  &  Requirements  Definition 
Verification  &Validation 

7.  How  will  you  structure  your 

Implementation  &  Transition 

program  to  meet  requirements 

and  mitigate  risk? 

2 

Integration 

8.  What  is  your  business-based 
transition  plan  that  meets 
customer  approval? 

4 

Configuration  Management 

Data  Management  &  Interface  Management 
Transition 

63 


Though  the  majority  of  participant  thought  the  questions  were  useful  about  30% 
identified  them  as  not  being  that  useful.  Several  participants  observed  if  you  can’t  answer 
these  you  are  not  managing  the  project  correctly.  The  first  three  seemed  to  be  the  most 
important.  The  least  important  question  was  identified  as  question  #  8.  This  is  likely 
because  not  all  SBIR  projects  transition  into  larger  projects. 

The  analysis  of  how  these  questions  map  back  to  the  DAG  processes  suggests 
these  questions  must  be  answered  to  successfully  manage  a  program.  However  different 
organizations  manage  SBIR  projects  which  value  and  implement  SE  differently.  The 
difference  in  opinion  from  participants  on  their  value  addresses  a  bigger  issue  of  the 
different  levels  of  understanding  and  appreciation  of  SE.  In  the  author’s  experience  two 
major  factors  are  responsible.  Either  workers  have  not  received  good  SE  education  and 
have  not  made  the  connection  that  good  SE  leads  to  good  program  management  or  they 
have  been  discouraged  from  their  experience  with  mismanaged  SE  efforts.  SE  must  be 
tailored  to  the  appropriate  level  for  a  program  or  project  to  ensure  it  is  value  added. 
Historically  the  DoD  had  tried  to  standardize  SE  processes  by  mandating  them.  This  one 
size  fits  all  approach  often  leads  to  an  increased  work  load  without  much  value  added  to 
the  program.  One  participant  identified  that  the  first  seven  questions  are  almost  offensive 
since  they  must  be  known.  The  author  is  not  surprised  how  an  experienced  project 
manager  can  see  it  this  way  however  the  management  of  SBIR  projects  varies  from 
project  managers  new  to  the  government  to  the  very  seasoned.  This  point  is  evident  in 
the  results  above  and  the  varying  understanding  of  the  questions. 
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AF  SEAM  Comparison 

The  author  compared  his  results  with  AF  SEAM  to  identify  the  applicable  SE 
tasks  for  the  SBIR  community.  AF  SEAM  and  AFRL  policy  do  not  directly  align  since 
AFRL  policy  maps  back  to  the  DAG.  Using  the  data  gathered  from  this  research  the 
author  reviewed  each  AF  SEAM  task  with  the  data  collected  and  translated  those  results 
into  AF  SEAM  SE  tasks.  Results  of  this  comparison  are  captured  in  Appendix  2. 

This  information  will  be  very  useful  to  the  Air  Force  SBIR  community  when  the 
current  draft  revision  of  63-1201  is  published  since  it  is  projected  to  align  with  Air  Force 
SEAM.  It  can  be  used  to  explain  what  is  applicable  to  the  SBIR  community  from  AF 
SEAM  and  also  illustrates  that  AF  SEAM  is  not  tailored  specific  for  the  SBIR 
community.  Only  about  50%  of  the  tasks  from  AF  SEAM  were  found  to  be  applicable 
for  the  SBIR  community.  Many  of  the  tasks  identified  in  AF  SEAM  are  not  applicable 
until  later  phases  of  a  program. 

These  findings  show  that  it  would  not  be  useful  to  implement  AF  SEAM  within 
the  SBIR  community.  In  addition  to  only  half  the  tasks  being  applicable,  AF  SEAM 
requires  a  large  manpower  effort  to  complete  it  due  to  its  190  SE  tasks.  Since  SBIR 
projects  do  not  have  resources  to  support  such  a  significant  SE  effort  and  it  is  not  tailored 
for  SBIR  a  more  tailored  approach  is  would  be  a  much  better  use  of  resources. 
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Research  Questions  Answered 


1 .  How  well  do  SBIR  projects  currently  implement  Systems  Engineering  policy? 
Answer:  The  SBIR  community  does  not  believe  all  SE  tasks  in  AFRL  policy 
apply  to  their  projects  and  does  consistently  implement  SE  tasks. 

2.  How  are  SE  Policies  implemented? 

Answer:  The  results  identified  a  wide  spectrum  of  interpretation  that  partially 
rests  on  the  sponsoring  organization  type  and  the  SBIR  phase.  Those  results 
identified  weak  areas  within  the  current  policy. 

3.  How  do  DoD  and  Air  Force  SE  processes  apply  to  SBIR  projects? 

Answer:  DoD  and  Air  Force  SE  processes  do  apply  to  SBIR  projects. 
However  they  must  be  tailored  for  the  scope  of  the  project.  Applicable  SE 
tasks  SBIR  projects  were  identified  in  Chapter  IV. 

4.  To  what  level  of  rigor  does  each  SE  process  apply  to  SBIR  projects? 

Answer:  The  number  of  tasks  for  each  SE  process  area  varies.  Specific  tasks 
for  each  area  where  captured  in  interview  results  and  comparison  of  AF 
SEAM  applicable  tasks. 
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5.  What  is  the  best  way  to  implement  these  processes? 

Answer:  The  best  way  to  implement  SE  to  a  SBIR  project  is  establish  a 
tailored  approach  and  follow  the  key  steps  outlined  in  policy.  SE  guidance  for 
this  is  identified  in  the  AFRL  SE  Guide.  Project  managers  must  have  a  good 
understanding  of  SE  and  tailor  a  solid  approach  for  their  project.  Project 
officers  can  use  results  from  this  research  as  a  guide  to  better  understand  what 
level  of  rigor  typically  applies  to  a  SBIR  project. 


Summary 

Results  from  the  literature  review  indentified  a  number  of  SE  processes  that  were 
applied  effectively  to  AFRL  projects.  The  interview  data  identified  what  SE  tasks  are 
being  accomplished  and  to  what  level  of  rigor  within  the  SBIR  community.  The  results 
identified  in  many  areas  that  the  SE  tasks  defined  in  policy  are  either  not  applicable  or  are 
not  being  accomplished  within  the  SBIR  community.  Better  SE  education  will  help 
project  managers  fully  tailor  SE  to  their  projects  and  ensure  applicable  tasks  are  being 
incorporated.  Tasks  that  are  not  applicable  however  should  not  be  required.  The  results 
from  the  literature  reviews  and  interviews  identified  those  tasks  and  can  be  used  to  better 
tailor  an  organization  s  SE  approach  for  SBIR  projects  to  avoid  wasting  resources  on  non 
value  added  tasks.  Overall  these  findings  again  illustrated  the  unique  nature  of  SBIR 
projects. 
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V.  Conclusions  and  Recommendations 


Chapter  IV  identified  the  SE  tasks  that  are  applicable  to  the  SBIR  community. 

The  case  studies  demonstrated  successfully  tailoring  a  SE  approach  for  S&T  projects. 
Interview  results  provided  applicable  SE  tasks  from  AFRL  policy  for  SBIR  projects.  The 
author  translated  those  results  with  his  SE  expertise  to  identify  what  SE  tasks  are 
applicable  for  SBIR  from  Air  Force  SEAM  in  Appendix  2.  Analysis  of  AFRL  SE  tasks 
and  AF  SEAM  identified  neither  of  them  are  specific  enough  for  SBIR.  Many  of  the 
tasks  were  not  applicable  during  SBIR  phase  I  and  II  projects  due  to  the  unique  nature  of 
SBIR  which  includes  limited  budget,  short  schedule  and  a  limited  scope.  Projects  also 
vary  greatly  across  the  Air  Force  and  DoD.  This  study  concludes  that  current  policy  does 
not  fully  define  SE  for  the  SBIR  community  and  that  SE  is  being  implemented  at  various 
level  amongst  the  different  organizations  that  manage  SBIR  projects. 

Results  from  this  study  also  identified  that  overall  the  SBIR  community  was  well 
educated  and  understood  how  certain  SE  processes  applied  to  their  projects.  However  the 
results  also  identified  that  they  are  very  week  in  many  areas.  I  believe  there  is  a 
misconception  within  the  community  that  some  areas  of  SE  do  not  apply  to  their  projects. 
All  areas  of  SE  apply  with  different  levels  of  rigor  for  any  project.  Leadership  and 
project  managers  must  ensure  adequate  levels  of  SE  are  being  incorporated  into  their 
projects  to  improve  their  chance  of  success,  limit  cost  and  schedule  overruns  and  meet 
perfonnance  goals.  Failure  to  follow  established  SE  processes  in  any  one  area  can  have 
significant  negative  consequences  to  the  project. 
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SBIR  SE  Checklist 


The  following  checklist  will  ensure  adequate  levels  of  SE  are  being  incorporated 
for  SBIR  projects.  This  SBIR  SE  Checklist  is  a  guide  for  project  managers  and  engineers 
to  ensure  all  SE  areas  are  adequately  addressed.  This  checklist  was  developed  using  the 
results  derived  from  this  research  and  align  with  the  10  AF  SEAM  SE  processes  areas. 

•  Represents  general  AF  SE  process  tasks  tailored  for  SBIR 
Represents  specific  SE  tasks  captured  in  analysis 


Table  25:  SBIR  SE  Checklist 


Requirements 

•  Determine  requirements  to  include  stakeholder  needs,  expectations, 
constraints,  and  interface  requirements. 

Translate  all  stakeholder  needs  to  technical  requirements. 
Requirements  made  quantifiable,  have  unique  definitions,  and 
specified  thresholds  and  objectives. 

Work  with  stakeholders  to  refine  requirements. 

Performance  parameters  and  constraints  allocated  and  derived 
technical  requirements  defined. 

Maintain  the  traceability  of  all  requirements  from  needs. 

Document  changes  and  record  rationale  of  changes. 

Project  Planning 

•  Identify  project  milestones  to  include  cost,  schedule  and  technical 
milestones. 

Define  the  scope  of  the  tech  effort  required  to  achieve  program 
technical  goals. 

Define  exit  criteria  and  products/deliverables  which  can  be  tracked 
with  progress  measured. 

Risk 

Management 

•  Develop  a  risk  management  plan  and  identify,  analyze,  identify 
handling  options,  mitigate  and  track  risk. 

Decision 

Analysis 

•  Establish  selection  criteria,  identify  &  evaluate  alternatives  and 
select  solution. 

Criteria  selected  for  decision  &  methods  to  be  used  in  conducting  the 
analysis. 

Identify  analysis  methods  and  conduct  analysis  of  alternatives. 

Design 

•  Establish  the  design  and  integration  baseline. 

Incorporate  the  lower-level  system  elements  into  a  higher-level 
system  element  in  the  physical  architecture. 

Indentify  constraints  &  interfaces  that  will  result  in  derived  technical 
requirements. 
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Technical 
Management  & 
Control 

•  Establish  and  maintain  the  project  environment,  integrated  product 
teams  (IPT),  measurements  approach  and  monitor  technical  reviews, 
work  products,  project  data,  corrective  actions  and  technical 
milestones. 

Measure  technical  progress,  technology  maturity  and  the 
effectiveness  of  plans  and  requirements. 

Demonstrate  and  confirm  completion  of  required  accomplishments 
and  project  exit  criteria. 

Configuration 

Management 

•  Establish  the  technical  baseline,  track  and  document  changes. 

Maintain  record  of  all  configurations  to  include  hardware,  software 
and  test  set  up  and  document  changes. 

Define  internal  and  external  interfaces. 

Project  data  managed  through  the  Defense  Technical  Information 
Center  (DTIC). 

Verification  & 
Validation 

•  Establish  and  maintain  the  overall  verification  strategy  and  plan  to 
include  verification  and  validation  criteria  and  an  integrated  testing 
approach  when  applicable.  Verify  and  Validate  that  the  project  has 
meets  the  required  parameters. 

Confirm  project  meets  design  specifications. 

Test  the  system  elements  against  their  defined  requirements. 

Test  the  performance  of  the  technology  against  the  original  program 
goals. 

Transition, 
Fielding,  & 
Sustainment 

•  Indentify  future  transition,  fielding,  &  sustainment  requirements  as 
needed  to  proceed  to  the  next  phase  of  the  project. 

Needs  of  follow-on  phases  considered  early  in  the  program  and 
included  in  all  of  the  technical  management  processes. 

Yield  the  fundamental  capability  of  the  program. 

Manufacturing 

•  Identify  and  maintain  documentation  relevant  to  the  future 
production  of  the  project. 

Develop  supporting  documentation  for  the  system. 

Project  Managers  using  this  checklist  will  begin  to  accomplish  some  of  these 
tasks  in  Phase  1  with  the  emphasis  of  demonstrating  the  project  is  feasible,  identifying 
stakeholders  and  defining  requirements.  By  the  end  of  Phase  2  all  of  the  above  tasks 
should  have  been  tailored  and  accomplished  for  the  project.  Projects  that  enter  Phase  2.5 
will  emphasize  on  further  defining  and  documenting  information  and  demonstrating  the 
technology  with  the  hopes  to  aid  in  the  future  transition  of  the  project  to  the  Phase  III. 
Phase  III  is  the  commercialization  phase. 
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Significance  of  Research 


This  research  identified  the  SE  policy  gap  in  the  SBIR  community  and  defined  the 
applicable  SE  tasks.  This  highlights  a  huge  risk  as  millions  of  dollars  are  spent  within  the 
DoD  each  year  on  SBIR  projects.  Failure  to  implement  good  SE  principles  can  and  will 
lead  to  cost  overruns,  schedule  slips  and  perfonnance  short  falls.  Findings  from  this 
research  should  be  used  to  tailor  a  SE  approach  for  SBIR  projects  to  ensure  SE  practices 
are  being  implemented  in  a  best  practice  manner. 

Recommendations  for  Action 

1 .  Organizational  policy  needs  to  be  tailored  for  SBIR.  The  SBIR  community 
should  use  the  identified  SBIR  SE  applicable  tasks  from  this  study  to  develop 
adequate  policy  and  SE  tasks  for  their  SBIR  projects.  SE  experience  varies 
greatly  amongst  the  SBIR  project  manager  within  the  DoD.  You  may  have  a 
project  manager  with  30  years  of  experience  or  a  newly  commissioned  officer 
managing  the  project.  There  for  it  is  critical  to  have  adequate  SE  policy  and 
guidance  in  place  to  ensure  that  critical  tasks  are  being  accomplished. 

2.  The  SBIR  community  should  incorporate  a  tailored  SE  approach  for 
their  projects.  The  approach  should  be  consistent  with  the  Streamline  SE 
Approach  for  the  S&T  community  outlined  in  the  draft  AFRL  SE  Guide.  The 
case  study  review  for  this  research  identified  the  benefits  of  using  such  an 
approach. 
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3.  The  SBIR  community  should  ensure  the  project  managers  receive 
adequate  SE  education  to  enable  them  to  tailor  SE  to  their  projects.  As 

the  scope  of  SBIR  projects  can  vary  greatly  it  can  be  challenging  for  project 
managers  to  understand  how  all  areas  of  SE  apply  to  their  projects.  Results 
from  the  interviews  also  identified  weak  areas  in  the  community  as  well  as 
misconceptions  that  some  areas  don’t  apply  to  them.  That  is  why  good  SE 
education  is  critical  for  project  managers  to  truly  make  the  connection  of  how 
SE  applies  to  their  projects.  The  Air  Force  and  DoD  have  many  good 
resources  to  provide  SE  education  to  the  work  force  such  as  DAU  and  AFIT. 
Supervisors  should  ensure  they  are  requiring  their  folks  to  take  advantage  of 
these  opportunities  and  are  continuing  to  develop  their  project  manager  skills. 

Recommendation  for  Future  Research 

The  research  reveled  several  opportunities  for  future  work  that  was  not  within  the 
scope  of  the  research. 

1 .  Good  SE  practices  are  often  hard  to  measure.  Further  data  could  be  collected 
from  each  of  the  SBIR  organization  managing  projects  to  identify  the 
successful  transition  rate  of  their  projects  and  the  SE  rigor  being  implemented 
within  the  organizations.  This  would  likely  illustrate  the  impact  of 
implementing  good  SE  practices  into  the  SBIR  community.  Transition  data 
was  not  available  for  this  research  project. 
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Appendix  1  -  SBIR  SE  Interview 


Small  Business  Innovative  Research  (SBIR)  Technical  Management  Processes  Interview 

Date: 

Survey  Directions:  Please  answer  each  question  in  regards  to  the  SBIR  projects  you  have  managed. 

Experience  Questions 

1.  What  is  your  current  job  title?  (Circle  one) 

Program  Manager  Project  Engineer.  Chief  Engineer  Other _ 

2.  What  AFRL  Directorate  do  you  work  for?  (Circle  one) 

RX  RY  RZ  RB  RH  RD 

RV  RW  711 HPW  AFOSR  Other _ 

3.  How  are  you  employed?  (Circle  one) 

Military  Civilian  Contractor 

Other _ 

4.  How  many  years  of  experience  do  you  have  in  your  job?  (Circle  one) 

1-2  3-5  5-10  10+ 

5.  What  level  of  APDP  certification  have  you  accomplished?  (Circle  one) 

123  none 

6.  In  what  APDP  area?  (Circle  one) 

PM  SPRDE  Other _ 

7.  What  phase  of  SBIR  projects  and  how  many  have  you  managed? 

Phase  I  _  Phase  II _  Other _ 
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8.  What  kinds  of  Stakeholders  Requirements  Definition  do  you  accomplish  on  average  for  your  SBIR 
projects?  Check  all  that  are  accomplished: 

o  All  inputs  from  relevant  stakeholders  translated  into  technical  requirements, 
o  Requirements  made  quantifiable,  have  unique  definitions,  and  specified  thresholds  and 
objectives. 

o  Work  with  the  user  to  establish  and  refine  goals,  attributes,  performance  parameters, 
and  financial  and  schedule  constraints,  and  then  ensure  that  all  relevant  requirements 
are  addressed  during  the  science  and  technology  effort, 
o  Translate  the  "customer  needs"  into  S&T  program  and  system  requirements, 
o  Other _ 

9.  What  kinds  of  Requirements  Analysis  do  you  accomplish  on  average  for  your  SBIR  projects? 

Check  all  that  are  accomplished: 

o  Obtain  sets  of  logical  solutions  to  improve  understanding  of  the  defined  requirements 
and  the  relationships  among  the  requirements  (e.g.  functional,  behavioral,  temporal), 
o  Performance  parameters  and  constraints  allocated  and  derived  technical  requirements 
defined. 

o  Partition  the  technical  problem  into  self-contained,  cohesive,  logical  groupings  of 
elements  and,  where  appropriate,  defined  the  key  interfaces, 
o  Other _ 

10.  What  kinds  of  Requirements  Management  do  you  accomplish  on  average  for  your  SBIR  projects? 
Check  all  that  are  accomplished: 

o  Maintain  the  traceability  of  all  requirements  from  needs 
o  Document  all  changes  to  those  requirements 
o  Record  the  rationale  for  those  changes. 

o  Traceable  to  some  current  or  potential  future  military  capability  need, 
o  Other _ 

11.  What  kinds  of  Decision  Analysis  do  you  accomplish  on  average  for  your  SBIR  projects?  Check  all 
that  are  accomplished: 

o  Criteria  selected  for  decision  and  methods  to  be  used  in  conducting  the  analysis, 
o  Analysis  conducted  to  help  choose  among  alternatives  to  achieve  a  balanced, 
supportable,  robust,  and  cost  effective  program, 
o  Analysis  methods  include  some  of  the  following:  trade  studies,  modeling  and 
simulation,  cost/benefit  analysis,  and  the  analytic  hierarchy  process  (AHP). 
o  Studies  are  augmented  with  virtual  and/or  physical  prototypes,  where  applicable,  prior 
to  making  decisions  on  best  alternative. 

o  Other _ 
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12.  What  kinds  of  Technical  Planning  do  you  accomplish  on  average  for  your  SBIR  projects?  Check  all 
that  are  accomplished: 

o  Technical  Planning  made  to  ensure  that  the  technical  activities  were  conducted  properly 
throughout  the  system's  life  cycle. 

o  Define  the  scope  of  the  technical  effort  required  to  achieve  program  technical  goals 
which  includes  exit  criteria  and  products/deliverables  which  can  be  tracked  with 
progress  measured. 

o  Indentify  constraints  and  interfaces  that  will  result  in  derived  technical  requirements. 

o  Contribute  input  to  the  Systems  Engineering  Plan,  which  is  owned  and  maintained  by 
the  acquisition  activity. 

o  Other _ 

13.  What  kinds  of  Technical  Assessment  do  you  accomplish  on  average  for  your  SBIR  projects?  Check 
all  that  are  accomplished: 

o  Measure  technical  progress,  technology  maturity,  and  the  effectiveness  of  plans  and 
requirements.  Activities  include:  Technical  Performance  Measurement,  Technology 
Readiness  Assessment,  and  the  conduct  of  technical  reviews. 

o  Demonstrate  and  confirm  completion  of  required  accomplishments  and  S&T  exit 
criteria. 

o  Discover  deficiencies  or  anomalies  that  may  result  in  the  application  of  corrective  action 
and  may  have  formed  the  technical  portion  of  a  continuous  process  improvement 
process  when  used  to  evaluate  application  of  SE  IAW  paragraph  2. 5. 5. 2. 

o  Technical  assessment  inputs  used  in  support  of  the  Laboratory  Management  Review 
process. 

o  Technical  assessment  activities  conducted  in  concert  with  existing  reviews  where 
possible  to  minimize  disruption  to  the  research  project. 

o  Other _ 

14.  What  kinds  of  Technical  Data  Management  do  you  accomplish  on  average  for  your  SBIR  projects? 
Check  all  that  are  accomplished: 

o  Project  data  managed  through  the  Defense  Technical  Information  Center  (DTIC) 

o  Or  similar  data  base. 

o  Other _ 
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15.  What  kinds  of  Risk  Management  do  you  accomplish  on  average  for  your  SBIR  projects?  Check  all 
that  are  accomplished: 

o  Develop  risk  management  plan  and  performed  the  following: 
o  Identified  risk 

o  Analyze  risk  and  define  probably  and  likelihood, 
o  Identify  handling  options 
o  Mitigate  risk 
o  Track  risk 

o  _ 


16.  What  kinds  of  Configuration  Management  do  you  accomplish  on  average  for  your  SBIR  projects? 

Check  all  that  are  accomplished: 

o  Ensure  the  repeatability  of  experimental  results  to  include  data  by  knowing  and  keeping 
a  record  of  the  laboratory  set-up  as  well  as  tracking  changes  to  it. 
o  Keep  a  record  of  laboratory  experimental  hardware  configuration  when  measurements 
are  gathered  including  such  things  as  calibration  status,  environmental  conditions, 
software  version  and  modifications  used,  and  documentation  (data)  resulting  from  the 
experiment/demonstration? 

o  A  complete  audit  trail  of  decisions  affecting  laboratory  equipment/software  design 
modifications  maintained. 

o  Other _ 

17.  What  kinds  of  Interface  Management  do  you  accomplish  on  average  for  your  SBIR  projects? 

Check  all  that  are  accomplished: 

o  Ensure  interface  definition  and  compliance  among  the  elements  that  compose  the 
laboratory  system  (internal  interfaces),  as  well  as  with  other  systems  with  which  the 
operational  system  or  system  elements  might  interact  (external  interfaces), 
o  Ensure  that  all  internal  and  external  interface  requirement  changes  are  properly 
documented  in  accordance  with  the  configuration  management  plan  and 
communicated  to  all  affected  elements  of  the  program, 
o  All  interfaces  defined  in  sufficient  detail  to  facilitate  necessary 

communication/interaction  among  system,  subsystem,  and  components, 
o  Other _ 
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18.  What  kinds  of  Architectural  Design  do  you  accomplish  on  average  for  your  SBIR  projects?  Check 
all  that  are  accomplished: 

o  Translate  the  outputs  of  the  Stakeholder  Requirements  Definition  and  Requirements 
Analysis  processes  into  alternative  technical  solutions  and  selects  a  final  technical  path 
to  explore. 

o  Iterate  Stakeholder  Requirements  Definition,  Requirements  Analysis,  and  with  the 
technical  management  processes  to  identify  and  select  the  best  solution  by  first 
developing  a  high-level  view  of  the  system  architecture  capable  of  meeting  stakeholder 
needs. 

o  Output  the  design  functional  or  physical  architecture  sufficiently  detailed  to  allow 
upward  and  downward  traceability  of  requirements, 
o  Generate  some  of  the  following:  AV-1,  SV-1,  OV-1  or  additional  DODAF  2.0  views 
o  Other _ 

19.  What  kinds  of  Implementation  do  you  accomplish  on  average  for  your  SBIR  projects?  Check  all 
that  are  accomplished: 

o  Yield  the  fundamental  capability  of  the  program. 

o  Include  some  testing  of  the  individual  elements  before  they  passed  to  Integration, 
o  Develop  supporting  documentation  for  the  system;  such  as  the  as-built  configuration,  or 
discovered  limitations  of  the  concept;  is  also  a  part  of  the  implementation  process, 
o  Other _ 

20.  What  kinds  of  Integration  do  you  accomplish  on  average  for  your  SBIR  projects?  Check  all  that 
are  accomplished: 

o  Incorporate  the  lower-level  system  elements  into  a  higher-level  system  element  in  the 
physical  architecture. 

o  Define  the  plan  or  strategy  for  the  Integration  process,  including  the  assembly 
sequence,  that  may  have  imposed  constraints  on  the  design  solution 
o  Other _ 

21.  What  kinds  of  Verification  do  you  accomplish  on  average  for  your  SBIR  projects?  Check  all  that 
are  accomplished: 

o  Confirm  that  the  laboratory/experimental  system  element  meets  design  specifications, 
o  Test  the  system  elements  against  their  defined  requirements  (predicted  versus 
experimental  results). 

o  Design  solutions  at  all  levels  of  the  physical  architecture  were  verified  through  a  cost- 
effective  combination  of  analysis,  examination,  demonstration,  and  testing,  all  of  which 
can  be  aided  by  modeling  and  simulation, 
o  Answer  the  verification  question  "Did  we  build  the  thing  right?". 

o  Other _ 
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22.  What  kinds  of  Validation  do  you  accomplish  on  average  for  your  SBIR  projects?  Check  all  that  are 
accomplished: 


o  Test  the  performance  of  the  technology  against  the  original  program  goals. 

o  Capture  any  testing  results/data  so  that  they  are  available  for  further 
development/research/maturation  efforts. 

o  Answer  the  validation  question  "Did  we  build  the  right  thing?". 

o  Other _ 

23.  What  steps  do  you  take,  or  tasks  do  you  perform,  to  help  with  the  possible  future  Transition  of 
your  SBIR  projects?  Check  all  that  are  accomplished: 

o  Deliver  a  supportable  technology  project  capable  of  being  put  in  the  hands  of  the 
warfighter. 

o  The  transition  process  applied  in  a  step-by-step  manner  to  move  the  technology  to  the 
next  level  in  the  developmental  cycle. 

o  Needs  of  follow-on  phases  considered  early  in  the  program  and  included  in  all  of  the 
technical  management  processes. 

o  Other _ 


Current  Practices  Questions 

24.  Do  you  use  the  8  Systems  Engineering  key  questions  identified  in  AFRLI  61-104,  and  listed  below, 
to  support  the  management  of  your  SBIR  project? 


Yes 

or 

No 

25.  Did  you  know  they  existed  prior  to  this  survey? 

Yes 

or 

No 

If  no  skip  to  question  28. 

26.  On  a  scale  from  1  to  7  overall  do  you  find  the  AFRL  8  Systems  Engineering  key  questions 
indentified  in  AFRLI  61-104  to  be  value  added  for  managing  your  SBIR  projects? 

1  2  3  4  5  6  7 

not  useful  a  little  useful  somewhat  useful  moderately  useful  useful  very  useful  extremely  useful 

Please  explain  in  short  detail: 
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27.  How  useful  do  you  find  each  of  the  8  SE  key  questions  for  management  of  your  SBIR 
projects?  Please  score  each  question  below  using  the  scale  provided.  (Circle  one) 

1  2  3  4  5  6  7 

not  useful  a  little  useful  somewhat  useful  moderately  useful  useful  very  useful  extremely  useful 

Score 

9.  Who  is  your  customer?  _ 

10.  What  are  the  customer's  requirements?  _ 

11.  How  will  you  demonstrate  you  have  met  the  requirements?  _ 

12.  What  are  the  technology  options?  _ 

13.  Which  is  the  best  approach?  _ 

14.  What  are  the  risks  to  developing  the  selected  technology?  _ 

15.  How  will  you  structure  your  program  to  meet  requirements  and  mitigate  risk? _ 

16.  What  is  your  business-based  transition  plan  that  meets  customer  approval?  _ 


28.  Which  of  the  AFRL  8  SE  key  questions  are  most  useful? 
Please  explain  in  short  detail: 


29.  Which  of  the  AFRL  8  SE  key  questions  are  least  useful? 

Please  explain  in  short  detail: 

30.  Is  there  a  Technical  Management  Process  Area  that  is  not  addressed  in  the  8  key 

questions  that  you  feel  should  be  incorporated?  Yes  or  No 

If  so,  please  explain  in  short  detail: 

31.  On  a  scale  from  1  to  7  in  your  overall  opinion  how  much  Systems  Engineering  rigor 
should  be  applied  to  SBIR  projects? 

1  2  3  4  5  6  7 

none  very  little  moderate  detail  in  good  detail  well  documented  very  detailed  extreme  detail 

Please  explain  in  short  detail: 


32.  Is  there  anything  else  you  would  like  to  add  for  his  survey? 
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Appendix  2:  SBIR  AF  SEAM  and  AFRL  Policy  SE  Tasks 


The  below  table  captures  the  applicable  SE  tasks  from  AF  SEAM.  It  illustrates 
that  many  tasks  from  AF  SEAM  apply  to  SBIR  however  many  do  not.  Approximately 
only  50%  of  AF  SEAM  tasks  were  found  to  be  either  applicable  in  most  cases  or 
sometimes  applicable  by  the  author.  The  author  was  able  to  translate  the  interview 
results  and  comments  using  his  expertise  as  a  SE  instructor  to  identify  the  applicable  AF 
SEAM  tasks  for  the  SBIR  community 

In  the  table  below,  “Applicable”  tasks  represent  that  50%  or  greater  of 
participants  identified  it  as  a  valid  SBIR  task.  “May  be  applicable”  tasks  represent  that 
25%  to  50  %  of  participants  identified  it  as  a  valid  SBIR  task.  “Typically  not  applicable” 
tasks  represent  less  25%  of  participants  identified  it  as  a  valid  SBIR  task. 
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Results  identified  a  limited  number  of  SE  tasks  performed  in  the  SBIR 
community  for  Configuration  Management.  Reviewing  SEAM  it  was  evident  that  most 
of  the  SEAM  tasks  are  for  a  mature  acquisition  program  and  not  the  SBIR  environment. 

Table  25:  AF  SEAM  SE  Tasks  for  SBIR  Projects 


AF  SEAM 

Configuration  Management  (CM) 

SG  ID 

AF  SEAM  Specific  Goal  (SG)  Title 

CMG1 

The  approach  for  technical  baseline  management  is  defined  and  documented. 

Applicable 

CMG1P1 

Identify  accountability  for  the  disposition  of,  access  to,  release  of  and  control 
of  the  technical  baselines. 

Typically  not 
applicable 

CMG1P2 

Establish  and  maintain  plans  for  managing  the  configuration  of  the  product. 

Typically  not 
applicable 

CMG2 

Establish  and  maintain  technical  baselines  while  managing  change 

Applicable 

CMG2P1 

Identify  the  configuration  items  and  related  work  products  that  will  be  placed 
under  configuration  management. 

Typically  not 
applicable 

CMG2P2 

Establish  and  maintain  configuration  and  change  management  systems. 

Typically  not 
applicable 

CMG2P3 

Create  or  release  technical  baselines. 

Typically  not 
applicable 

CMG2P4 

Track  and  control  changes. 

Applicable 

CMG3 

Integrity’  of  baselines  is  established  and  maintained 

Applicable 

CMG3P1 

Establish  and  maintain  records  describing  configuration  items 

May  be  applicable 

CMG3P2 

Perform  configuration  audits  to  maintain  integrity  of  the  configuration 
baselines 

Typically  not 
applicable 

Results  identified  several  SE  tasks  performed  in  the  SBIR  community  that 
translated  well  with  Air  Force  SEAM.  Those  tasks  included  selecting  criteria  to  be  used, 
methods  to  be  used  and  conducting  analysis  of  alternatives. 


Decision  Analysis  (DA) 

SG  ID 

AF  SEAM  Specific  Goal  (SG)  Title 

DAG1 

Base  decisions  on  an  evaluation  of  alternatives  using  established  criteria 

Applicable 

DAG1P1 

Establish  and  maintain  guidelines  to  determine  which  issues  are  subject  to  a 
formal  evaluation  process 

May  be  applicable 

DAG1P2 

Establish  and  maintain  the  criteria  for  evaluating  alternatives,  the  relative 
ranking  of  these  criteria,  and  select  the  evaluation  methods 

Applicable 

DAG1P3 

Identify  alternative  solutions  to  address  issues 

Applicable 

DAG1P4 

Evaluate  alternative  solutions  using  the  established  criteria  and  methods 

Applicable 

DAG1P5 

Select  the  solution(s)  from  the  alternatives  and  document  decisions  based  on 
the  evaluation 

Applicable 
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Results  identified  a  limited  number  of  SE  tasks  performed  in  the  SBIR 
community  for  Design.  Design  from  AFRL  policy  included  Architecture,  Interface 
Management  and  Integration  Tasks.  Results  from  the  interview  identified  a  low 
percentage  of  SE  tasks  in  these  areas  being  completed. 


Design  (D) 

SG  ID 

AF  SEAM  Specific  Goal  (SG)  Title 

DG1 

The  design  is  based  upon  a  documented  architecture,  traceable  to 
requirements,  and  optimized  for  the  set  of  requirements  and  constraints 

Typically  not 
applicable 

DG1P1 

Establish  and  maintain  the  architectural  design  baseline 

Typically  not 
applicable 

DG1P2 

Establish  and  maintain  interface  designs 

Typically  not 
applicable 

DG1P3 

Establish  and  maintain  design  artifacts  that  describe  the  conditions,  functions, 
operating  modes,  and  operating  states  specific  to  the  components  of  the 
architecture 

Typically  not 
applicable 

DG1P4 

Develop  potential  product-component  solutions,  alternatives,  and  selection 
criteria 

Typically  not 
applicable 

DG1P5 

Analyze  and  select  product-component  solutions  that  best  satisfy  the 
established  criteria 

Typically  not 
applicable 

DG2 

Develop  and  document  a  detailed  design  and  implementation  strategy 

Typically  not 
applicable 

DG2P1 

Establish  initial  product-component  designs  and  development  strategies 

Typically  not 
applicable 

DG2P2 

Evaluate  whether  the  product-components  should  be  developed,  purchased,  or 
reused  based  on  established  criteria 

Typically  not 
applicable 

DG2P3 

Establish  detailed  designs  for  the  product-component 

Typically  not 
applicable 

DG2P4 

Establish  and  maintain  a  technical  data  package 

Typically  not 
applicable 

DG3 

Assemble  the  design/development  prototype(s)  in  accordance  with  the  detailed 
design  and  integration  strategy’ 

May  be  applicable 

DG3P1 

Establish  and  maintain  the  product  integration  approach 

May  be  applicable 

DG3P2 

Establish  and  maintain  procedures  and  criteria  for  integration  of  the  product- 
components 

May  be  applicable 

DG3P3 

Manage  internal  and  external  interface  definitions,  designs,  and  changes  for 
products  and  product-components 

May  be  applicable 

DG3P4 

Conduct,  prior  to  assembly  product-component  verification 

May  be  applicable 

DG3P5 

Assemble  product-components  according  to  the  product  integration  sequence 
and  established  procedures 

Typically  not 
applicable 
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Results  identified  several  Implementation  SE  tasks  performed  in  the  SBIR 
community.  Implementation  is  captured  under  Manufacturing  in  AF  SEAM.  However 
those  tasks  identified  by  AF  SEAM  do  not  identify  with  the  SBIR  community  as  seen 
below  since  little  from  the  AFRL  Tasks  could  be  translated  into  SEAM  tasks  as  seen 
below. 


Manufacturing  (M) 

SG  ID 

AF  SEAM  Specific  Goal  (SG)  Title 

MGl 

Prepare  for  manufacturing 

May  be  applicable 

MG1P1 

Establish  and  maintain  strategy  and  plans  for  manufacturing 

Typically  not 
applicable 

MG1P2 

Perform  concurrent  design  and  manufacturing  engineering 

May  be  applicable 

MG1P3 

Establish  and  maintain  manufacturing  technical  data 

Typically  not 
applicable 

MG2 

Transition  from  development  to  repeatable  and  economical  production  at 
desired  rate 

Typically  not 
applicable 

MG2P1 

Establish  and  maintain  plans  for  transition  to  production 

May  be  applicable 

MG2P2 

Qualify/proof  manufacturing  processes,  special  tools  and  test  equipment 

Typically  not 
applicable 

MG2P3 

Ensure  readiness  for  manufacturing 

Typically  not 
applicable 

MG3 

Manufacture  the  product  in  accordance  with  plans  and  specifications 

Typically  not 
applicable 

MG3P1 

Ensure  that  production  at  desired  rates  is  conducted  according  to  the  plan 

Typically  not 
applicable 

MG3P2 

Establish  and  maintain  inventory  and  supplier  management/control 

Typically  not 
applicable 

MG3P3 

Complete  First  Article  Inspection  (FAI) 

Typically  not 
applicable 

MG4 

Product  and  process  cpiality  is  assessed  and  improved 

Typically  not 
applicable 

MG4P1 

Establish  and  maintain  piece  part  control  and  perform  manufacturing 
screening 

Typically  not 
applicable 

MG4P2 

Establish  and  maintain  a  quality  management  system 

Typically  not 
applicable 

MG4P3 

Establish  and  maintain  defect  control 

Typically  not 
applicable 
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Results  for  Technical  Planning  identified  several  tasks  performed  in  the  SBIR 
community  that  translate  to  the  SE  tasks  captured  under  Project  Planning  in  AF  SEAM. 
Most  are  applicable  or  may  be  applicable  in  the  SBIR  community  as  illustrated  below. 


Project  Planning  (PP) 

SG  ID 

AF  SEAM  Specific  Goal  (SG)  Title 

PPG1 

Establish  and  maintain  estimates  of  project  planning  parameters 

Applicable 

PPG1P1 

Define  the  project  life  cycle  phases,  milestones,  and  key  decision  points 

Applicable 

PPG1P2 

Establish  a  work  breakdown  structure  (WBS)  to  organize  the  effort 

May  be  applicable 

PPG1P3 

Establish  and  maintain  the  scope  of  the  work  products  and  tasks  that  describe 
the  project  cost  and  schedule 

Applicable 

PPG1P4 

Establish,  validate,  and  maintain  estimates  for  cost  and  schedule 

Applicable 

PPG2 

Establish  and  maintain  integrated  plans 

May  be  applicable 

PPG2P1 

Assign  responsibility  for  acquisition  and  sustainment  management,  support, 
and  product  enhancement 

Typically  not 
applicable 

PPG2P2 

Establish  and  maintain  engineering  plans  to  accomplish  project 

Applicable 

PPG2P3 

Plan  for  the  management  of  project  data 

Applicable 

PPG2P4 

Plan  for  necessary  resources,  including  personnel  knowledge  and  skills,  to 
perform  the  project  tasks 

Applicable 

PPG2P5 

Plan  the  involvement  of  identified  stakeholders 

Applicable 

PPG2P6 

Establish  and  maintain  the  technology  development  strategy 

Applicable 

PPG2P7 

Plan  for  product  Reliability,  Availability,  and  Maintainability  (RAM) 

May  be  applicable 

PPG2P8 

Establish  and  maintain  an  Integrated  Master  Plan  and  Integrated  Master 
Schedule  (IMP/IMS) 

May  be  applicable 

PPG3 

Establish  and  maintain  commitment  to  the  technical  plan 

Applicable 

PPG3P1 

Review  all  plans  to  understand  commitments  and  ensure  the  technical  plans 
and  overall  plans  are  integrated  and  consistent 

May  be  applicable 

PPG3P2 

Reconcile  the  technical  plans  to  reflect  available  and  estimated  resources 

Applicable 

PPG3P3 

Obtain  commitment  from  relevant  stakeholders  responsible  for  performing  and 
supporting  execution 

Applicable 
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Results  for  Requirements  Definition,  Analysis  and  Management  identified  tasks 


performed  in  the  SBIR  community  that  translate  to  the  SE  tasks  captured  under 
Requirements  in  AF  SEAM.  Most  tasks  were  applicable  but  some  did  not  apply  for 
SBIR  projects  as  illustrated  below. 


Requirements  (R) 

SG  ID 

AF  SEAM  Specific  Goal  (SG)  Title 

RG1 

Stakeholder  needs,  expectations,  constraints,  and  interface  requirements  are 
collected  and  translated  into  a  definition  of  needed  product 
capabilities/characteristics  for  all  phases  of  the  life  cycle 

Applicable 

RG1P1 

Elicit  stakeholder  needs,  expectations,  constraints,  and  interfaces 

Applicable 

RG1P2 

Establish  and  maintain  concepts  of  operations  and  support  that  define  the 
operational  capability  required 

Applicable 

RG1P3 

Transform  stakeholder  needs,  expectations,  constraints,  and  interfaces  into  a 
prioritized  requirements  baseline 

Applicable 

RG1P4 

Establish  and  maintain  a  requirements/decision  data  archive  to  document 
requirements  and  related  technical  decisions 

Applicable 

RG2 

Requirements  are  refined,  elaborated,  and  allocated  to  support  design  or 
service(s) 

May  be  applicable 

RG2P1 

Establish  and  maintain  design  mission  reference  profiles  that  define  the 
product  characteristics  required  in  engineering  terms  and  document  the 
interaction  of  the  product  with  the  environment,  other  systems  and  operational 
users 

Typically  not 
applicable 

RG2P2 

Allocate  the  requirements  to  each  product-component 

Applicable 

RG3 

Iteratively  analyze  and  validate  operational  and  derived  requirements 
throughout  the  product  life  cycle 

Applicable 

RG3P1 

Analyze  requirements  to  ensure  that  they  are  necessary  and  sufficient 

Applicable 

RG3P2 

Analyze  requirements  to  balance  stakeholder  needs  and  constraints 

Applicable 

RG3P3 

Validate  requirements  to  ensure  the  evolving  product  will  perform  as  intended 
in  the  operational  environment 

Applicable 

RG4 

Requirements  are  managed  and  controlled,  and  inconsistencies  with  technical 
plans  and  work  products  are  identified 

Typically  not 
applicable 

RG4P1 

Use  a  disciplined  process  for  accepting,  vetting,  approving,  and  providing 
requirements  and  changes  to  the  suppliers 

Typically  not 
applicable 

RG4P2 

Establish  and  maintain  commitment  to  the  requirements 

Applicable 

RG4P3 

Establish  and  maintain  bidirectional  traceability  between  requirements  and 
work  products 

Applicable 

RG4P4 

Identify  and  resolve  inconsistencies  between  requirements,  project  plans,  and 
work  products 

Applicable 
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Results  identified  all  AF  SEAM  SE  tasks  apply  to  the  SBIR  community  for  Risk 


Management. 


Risk  Management  (RM) 

SG  ID 

AF  SEAM  Specific  Goal  (SG)  Title 

RMG1 

Prepare  for  Risk  Management 

Applicable 

RMG1P1 

Determine  risk  sources  and  categories 

Applicable 

RMG1P2 

Define  the  parameters  used  to  analyze  and  categorize  risks,  and  the 
parameters  used  to  control  the  risk  management  effort 

Applicable 

RMG1P3 

Establish  and  maintain  the  strategy  and  plans  to  be  used  for  risk  management 

Applicable 

RMG2 

Identify  and  analyze  risks  to  determine  their  relative  importance 

Applicable 

RMG2P1 

Identify  and  document  the  technical  risks 

Applicable 

RMG2P2 

Evaluate  and  categorize  each  identified  risk  using  the  defined  risk  categories 
and  parameters,  and  determine  its  relative  priority 

Applicable 

RMG3 

Perform  risk  handling  to  manage  adverse  impacts  on  the  project 

Applicable 

RMG3P1 

Establish  and  maintain  plans  for  mitigating  each  of  the  important  risks  to  the 
project 

Applicable 

RMG3P2 

Monitor  and  assess  risk  handling  activities 

Applicable 
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Results  identified  several  SE  tasks  applicable  for  Transition  in  the  SBIR 
Community.  However  none  of  these  tasks  translate  to  AF  SEAM  SE  tasks  for  Transition, 
Fielding,  &  Sustainment.  All  of  these  tasks  become  relevant  in  later  stages  of  technology 
development  and  not  during  SBIR  Phase  I  &  II. 


Transition,  Fielding,  &  Sustainment  (S) 

SG  ID 

AF  SEAM  Specific  Goal  (SG)  Title 

SGI 

Prepare  to  support  operations,  maintenance,  repair,  and  disposal  of  the 
product 

Typically  not 
applicable 

TFSG1P1 

Establish  and  maintain  plans  for  logistics  support  of  the  product 

Typically  not 
applicable 

TFSG1P2 

Establish  and  maintain  the  strategy  and  plan(s)  for  transitioning  acquired 
products  into  operational  use  and  support 

Typically  not 
applicable 

TFSG1P3 

Establish  and  maintain  plan(s)  for  the  disposal  of  the  product 

Typically  not 
applicable 

SG2 

Ensure  the  resources,  capacity  and  capability  to  support  the  operations, 
maintenance,  repair,  and  disposal  of  the  product  are  ready  prior  to  need 

Typically  not 
applicable 

TFSG2P1 

Establish  and  maintain  budgets  for  sustainment  activities 

Typically  not 
applicable 

TFSG2P2 

Establish  and  maintain  processes  and  procedures  for  repair,  overhaul,  or 
modification 

Typically  not 
applicable 

TFSG2P3 

Ensure  readiness  for  fielding  and  transition  to  operations  and  support 

Typically  not 
applicable 

TFSG2P4 

Ensure  product  support  is  maintained  during  transition 

Typically  not 
applicable 

TFSG2P5 

Establish  and  maintain  the  required  facilities,  manpower,  tooling  and  test 
equipment  for  repair,  overhaul,  or  modification 

Typically  not 
applicable 

SG3 

Repair,  overhaul,  or  modify  the  product 

Typically  not 
applicable 

TFSG3P1 

Repair,  overhaul  or  modify  the  product  in  accordance  with  established 
procedures  and  processes 

Typically  not 
applicable 

TFSG3P2 

Establish  and  maintain  inventory  and  supplier  management/control  to  execute 
the  repair,  overhaul  or  modification 

Typically  not 
applicable 

SG4 

Maintain  Operational  Safety,  Suitability,  and  Effectiveness  (OSS&E) 

Typically  not 
applicable 

TFSG4P1 

Establish  and  maintain  OSS&E  baseline(s) 

Typically  not 
applicable 

TFSG4P2 

Identify  and  monitor  safety  critical  items 

Typically  not 
applicable 

TFSG4P3 

Identify  and  mitigate  hazards 

Typically  not 
applicable 

TFSG4P4 

Identify  and  monitor  operations  and  maintenance  data 

Typically  not 
applicable 

TFSG4P5 

Execute  (as  required)  the  plan  for  decommissioning  and  disposal  of  the 
product 

Typically  not 
applicable 
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Results  for  Technical  Assessment  identified  tasks  perfonned  in  the  SBIR 
community  that  translate  to  the  SE  tasks  captured  under  Technical  Management  & 
Control  in  AF  SEAM.  Most  tasks  were  applicable  except  executing  supplier  agreements 
since  this  does  not  take  place  during  SBIR  Phase  I  &  II  but  would  be  relevant  after  the 
project  transitions. 


Technical  Management  &  Control  (TMC) 

SG  ID 

AF  SEAM  Specific  Goal  (SG)  Title 

TMCG1 

Prepare  for  Integrated  Management 

Applicable 

TMCG1P1 

Establish  and  maintain  the  project  environment 

Applicable 

TMCG1P2 

Establish  and  maintain  supplier  agreements 

Typically  not 
applicable 

TMCG1P3 

Establish  and  maintain  integrated  product  teams  (IPT) 

May  be  applicable 

TMCG1P4 

Establish  and  maintain  measurement  approach 

Applicable 

TMCG2 

Perform  Integrated  Management 

Applicable 

TMCG2P1 

Monitor  and  control  the  project  in  accordance  with  project  commitments 

Applicable 

TMCG2P2 

Monitor  &  control  coordination  and  collaboration 

Applicable 

TMCG2P3 

Execute  Supplier  Agreements 

Typically  not 
applicable 

TMCG2P4 

Obtain  and  analyze  specified  measurement  data 

Applicable 

TMCG2P5 

Monitor  the  development  and  delivery  of  project  data 

Applicable 

TMCG3 

Monitor  &  Control  Technical  Progress 

Applicable 

TMCG3P1 

Technical  reviews  and  audits  are  conducted  when  all  the  key  entry  criteria  are 
met  and  closed  when  the  exit  criteria  are  met 

Applicable 

TMCG3P2 

Assess  results  of  technical  reviews  to  support  key  milestone  decisions,  higher 
level  reporting,  and  project  re -planning  as  required 

Applicable 

TMCG3P3 

Manage  project  work  products  and  data 

Applicable 

TMCG4 

Monitor  &  Control  Corrective  Actions 

Applicable 

TMCG4P1 

Collect  and  analyze  the  project  issues  to  determine  and  track  corrective 
actions 

Applicable 

TMCG4P2 

Establish  and  maintain  a  deficiency  reporting  system 

Applicable 

TMCG4P3 

Manage  corrective  actions  to  closure 

Applicable 
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Results  for  Verification  and  Validation  identified  tasks  perfonned  in  the  SBIR 
community  that  translate  to  the  SE  tasks  captured  under  Verification  and  Validation  in 
AF  SEAM.  Most  tasks  were  applicable  but  some  did  not  apply  for  SBIR  projects  since  a 
limited  amount  of  testing  in  perfonned  during  SBIR  Phase  I  &  II.  Additional  testing  will 
take  place  when  the  project  transitions  and  those  tasks  will  be  applicable  then. 


Verification  and  Validation  (V) 

SG  ID 

AF  SEAM  Specific  Goal  (SG)  Title 

VG1 

Prepare  for  verification 

Applicable 

VG1P1 

Establish  and  maintain  the  overall  verification  strategy  and  plan,  including 
integrated  testing  approach 

Applicable 

VG1P2 

Identify  the  work  products  to  be  verified  and  the  verification  methods  that  will 
be  used 

Applicable 

VG1P3 

Establish  and  maintain  the  environment  and  resources  needed  to  support 
verification 

Applicable 

VG1P4 

Establish  verification  procedures  and  criteria 

Applicable 

VG2 

Peer  reviews  are  performed 

May  be  applicable 

VG2P1 

Prepare  for  peer  reviews  of  selected  work  products 

May  be 

Applicable 

VG2P2 

Conduct  peer  reviews  on  selected  work  products  and  identify  issues  resulting 
from  the  peer  review 

May  be 

Applicable 

VG3 

Work  products  are  veri  fied 

Applicable 

VG3P1 

Perform  verification  on  the  selected  work  products 

Applicable 

VG3P2 

Analyze  and  document  the  results  of  all  verification  activities 

Applicable 

VG3P3 

Initiate  and  document  corrective  actions 

Applicable 

VG4 

Prepare  for  validation 

Applicable 

VG4P1 

Develop  a  product  validation  strategy  and  identify  work  products  for 
validation 

Applicable 

VG4P2 

Establish  and  maintain  validation  criteria,  methods  and  procedures 

Applicable 

VG4P3 

Establish  and  maintain  the  environment  and  resources  needed  to  support 
validation 

Applicable 

VG4P4 

Ensure  appropriate  certifications  &  accreditations  have  been  completed 

Typically  not 
applicable 

VG4P5 

Establish  and  maintain  a  documented  plan  for  validation 

Applicable 

VG5 

Validate  product  to  ensure  that  it  will  be  sa  fe,  suitable  and  effective  in  the 
intended  operating  environment 

Typically  not 
applicable 

VG5P1 

Perform  validation  on  the  selected  products  and  product-components 

May  be  applicable 

VG5P2 

Analyze  and  document  the  results  of  the  validation  activities 

Applicable 
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Appendix  3:  SBIR  SETT  Tool  Output 


Research  of  past  work  on  this  topic  identified  a  prior  AFIT  thesis  titled  “A  Tailored 
Systems  Engineering  Framework  for  Science  and  Technology  Projects”  from  March 
2009.  This  thesis  built  a  tool  for  identifying  what  Systems  Engineering  principals  should 
by  applied  to  a  particular  program  in  AFRL  based  on  inputting  the  following  parameters 
identified  in  the  example  below: 

Parameters  inputted  into  tool: 

Welcome  to  the  Systems  Engineering  Tailoring  tool  for  Science  &  Technology  Projects  (SETT-I 

Please  select  the  following  project  discriminants  that  apply.  Hover  on  a  selection  to  see  a  description. 

(To  select  a  discriminant  place  a  "1"  in  the  cell  next  to  the  domain  value.  Leave  all  other  domain  values  mark 


RDT&E  Category 

6.1  (Basic  Research) 

=H 

6.2  (Applied  Research) 

6.3  (Advanced  Technology  Development)' 

Technology  Readiness  Level  | 

1-2 

0 

3-4 

5-6 

0 

7-9 

0 

Project  Budget 

<$500K 

0 

$500K - $2M 

>$2M 

0 

Integration  Level  | 

Subsystem 

System 

0 

Mission 

0 

Core  Process  1 

CP-1  (Far  Term) 

m 

CP-2  (Medium  Term) 

CP-3  (Urgent  User  Needs) 

Reauirements  Maturity 

Technology  Push 

Requirements  Pull 

After  selecting  your  project  discriminant  click  on  the  worksheet  tab  "Tailored  SE  Activities"  below. 
Drill  down  into  lower  level  SE  activities  by  using  the  marks  on  the  left  of  that  worksheet. 


Inputted  parameters  above  would  fit  some  SBIR  programs.  Below  is  the  output  for  the 
tool  for  those  parameters: 
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SE  Activity  ▼ 

Source 

Tools 

SE  Rigor 

TP-1  (Requirements  Development) 

25%  - 100% 

Establish  Communications  with  Stakeholders 

DAG  Sec4.2.4.1 

25%  - 100% 

Identify  Project  Constraints 

DAG  Sec  4. 2.4.1 

30%  - 100% 

Determine  Required  Capabilities 

DAG  Sec4.2.4.1 

25%  -  80% 

Determine  Desired  Performance 

DAG  Sec  4. 2.4.1 

30%  - 100% 

TP-2  (Logical  Analysis) 

0%  - 100% 

Analysis  Preparation 

DAG  Sec  4. 2.4.2. 

100%  - 100% 

Perform  Functional  Analysis 

DAG  Sec  4. 2.4.2. 

DoDAF  OV-5, 

SV-5 

60%  - 100% 

Perform  Behavioral  Analysis 

DAG  Sec4. 2.4.2. 

DoDAF  OV- 

6(a,b,c),  SV- 
10(a,b,c) 

80%  - 100% 

Perform  Environmental  Analysis 

DAG  Sec  4. 2.4.2. 

10%  - 100% 

Design  Factors  Analysis 

INCOSE  Pg  4.6 

0%  - 100% 

Develop  Functional  Architecture 

DAG  Sec4.2.4.2. 

30%  - 100% 

TP-3  (Design  Solution) 

10%  - 100% 

Define  Design  Problem 

Buede  Pg  31,  39 

20%  - 100% 

Generate  Alternative  Design  Solutions 

Buede  Pg  31,  39 

10%  - 100% 

Evaluate  Design  Alternatives 

DAG  Sec4.2.4.3 

20%  -  80% 

TP-4  (Implementation) 

5%  - 100% 

Generate  Implementation  Strategy 

INCOSE  Pg  4.10 

30%  - 100% 

Fabricate  Hardware 

DAG  Sec4.2.4.4. 

30%  -  80% 

Code  Software 

DAG  Sec4.2.4.4. 

30%  - 100% 

Conduct  Unit  Testing 

INCOSE  Pg  4.10 

30%  - 100% 

Conduct  Training 

INCOSE  Pg  4.10 

5%  -  5% 

Prepare  for  Integration 

DAG  Sec4.2.4.4. 

30%  -  80% 

TP-5  (Integration) 

10%  - 100% 

Determine  Integration  Process 

Buede  Pg  310 

80%  -  80% 

Conduct  Assembly  /  Integration  of  System 

INCOSE  Pg  4.12 

10%  - 100  % 

Relevant  Environment 

DAG  Sec4.2.4.5. 

10%  -  10% 

TP-6  (Verification) 

30%  - 100% 

Plan  Verification 

Buede  Pg  314 

30%  - 100% 

Execute  Verification 

INCOSE  Pg  4.14 

30%  - 100% 

TP-7  (Validation) 

30%  - 100% 

Plan  Validation 

Buede  Pg  314 

30%  - 100% 

Execute  Validation 

Buede  Pg51; 
INCOSE  Pg  4.17 

30%  - 100% 
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SE  Activity 

Source 

Tools 

SE  Rigor 

TP-8  (Transition) 

0%  - 100% 

Identify  Transition  Opportunities 

INCOSE  Pg  3.2 

50%  - 100% 

Qualify  Production  Item 

Buede  Pg  314 

0%  -  0% 

Execute  Transition 

INCOSE  4.15 

30%  -  100% 

TMP-1  (Decision  Analysis) 

100%  -  100% 

Identify  Strategy  for  Making  Decision 

INCOSE  Pg  5.8 

100%  -  100% 

Execute  Decision  Making  Strategy 

DAG  4.2.3. 1 

100%  -  100% 

TMP-2  (Technical  Planning) 

0%  -  100% 

Plan  Systems  Engineering 

DAG  Sec  4.2. 3. 2. 

0%  -  100% 

Implement  Technical  Plan 

INCOSE  Pg  8.1-13 

Integrated 

Master  Plan 

30%  -  100% 

Evaluate  Plan  to  Address  Needs 

INCOSE  Pg  3.8-9 

0%  -  0% 

TMP-3  (Technical  Assessment) 

70%  -  100% 

Prepare  for  Technical  Assessment 

DAG  Sec  4.2. 3. 3. 

80%  -  100% 

Perform  Technical  Assessment 

INCOSE  Pg  5.5 

70%  -  100% 

TMP-4  (Requirements  Management) 

5%  -  100% 

Determine  Roles/Responsibilities  During  Reqs  Generation 

Process 

Buede  Pg  129 

75%  -  100% 

Define  System  Capabilities  and  Performance  Objectives 

INCOSE  Pg  7.6-12 

30%  -  100% 

Validate  Requirements  Development  Process 

Buede  Pg  41 

25%  -  25% 

Ensure  Requirements  Feasibility  and  Validity 

Buede  Pg  40 

5%  -  95% 

Document  Requirements 

INCOSE  Pg  4.3, 

4.4 

25%  - 100% 

Ensure  Traceability  of  Requirements 

DAG  Sec  4.2. 3.4.; 
Buede  Pg  158; 
INCOSE  Pg  3.10 

25%  -  100% 

Establish  Process  for  Requirements  Changes 

Buede  Pg  129 

25% -80% 
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SE  Activity  w 

Source 

Tools 

SE  Rigor 

TMP-5  (Risk  Management) 

10%  - 100% 

Risk  Planning 

DAG  Sec  4.2.3. 5 

Risk  Management 

Framework 

0% 

Risk  Identification 

DAG  Sec  4.2.3.S.; 
Buede  Pg  314; 
INCOSE  Pg  5.10- 

5.11 

Documentation 

Reviews;  Information 
Gathering 

(Brainstorming,  Delphi 
Technique,  Interviews, 
SWOT  (Strength- 
Weakness-Opportunity- 
Threat)  Analysis) 

30%  - 100% 

Risk  Analysis  (Qualitative  &  Quantitative) 

DAG  Sec  4.2.3. 5.; 
Buede  Pg  382 

ARENA,  CORE,  MATLAB 

State  Flow  Modeler, 
Crystal  Ball  (Excel  add¬ 
in) 

f 

100%.  100% 

Risk  Handling 

DAG  Sec  4. 2.3.5. ; 
INCOSE  Pg  5.11 

100%-  100% 

Risk  Monitoring 

DAG  Sec  4.2.3. 5. 

80%  -  100% 

Risk  Documentation 

DAG  Sec  4.2.3.S.; 
INCOSE  Pg  5.10 

10%  -  100% 

TMP-6  (Configuration  Management) 

30%  - 100% 

Develop  Configuration  Baselines 

DAG  Sec  4.2.3. 6.; 
INCOSE  Pg  4.17 
INCOSE  Pg  5.12 

30%  -  100% 

Establish  Configuration  Change  Control  Plan 
(Establish  configuration  control  cycle  that 
incorporates  evaluation,  approval, 
validation,  and  verification  of  change 
requests) 

DAG  Sec  4.2.3. 6.; 
INCOSE  Pg  5.13 

30%  -  30% 

Develop  and  Maintain  Configuration  Control 
Documentation 

INCOSE  Pg  4.6  & 

5.13 

30%  -  30% 

Maintain  Configuration  Baselines 

DAG  Sec  4.2.3. 6.; 
INCOSE  Pg  4.12  Si 

5.13 

30%  -  100% 
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SE  Activity  Z. 

Source 

Tools 

SE  Rigor 

TMP-7  (Technical  Data  Management) 

10%  - 100% 

Develop  Data  Management  Plan 

DAG  Sec  4.2.3.7.; 
INCOSE  Pg  5.15 

Core  Architecture  Data 

Model 

30%  - 100% 

Determine  /  Define  System  Relevant 
Information 

INCOSE  Pg  5.15 

100%  -  100% 

Identify  System  Data  to  Purchase 

DAG  Sec  4.2.3. 7.1. 

100%  -  100% 

Determine  Data  Protection  Requirements 

DAG  Sec  4.2.3. 7. 2. 

100%  -  100% 

Address  Long-term  Data  Storage 
Requirements 

DAG  Sec  4.2.37.3. 

50%  -  50% 

Record  Program  Data 

INCOSE  Pg  4.10 

10%  -  100% 

Make  Project  Data  Available 

INCOSE  Pg  5.15 

50%  -  100% 

TMP-8  (Interface  Management) 

10% - 100% 

Define  Interface  Requirements  and  Control 

Methods 

Buede  Pg  294; 
INCOSE  Pg  4.8 

100%-  100% 

Develop  System  Interface  Control  Methods 

DAG  Sec  4.2.4.I. 

DAG  Sec  4.2.3.S.; 
Buede  Pg  50; 
INCOSE  Pg  4.7 

30%  - 100% 

Generate  Interface  Control  Documentation 

DAG  Sec  4.2.3.S. 

DAG  Sec  4.2A5. 

DoDAF  SV-1,  Interface 

Control  Document 

60%  -  100% 

Utilize  Interface  Controls 

DAG  Sec  4.2.3.8.; 
Buede  Pg  39 

f 

ia%  -  ioa% 

Fundamental  Principles  (Applicable 
to  ALL  PROCESSES) 

100%-  100% 

Utilize  Enterprise  Capabilities 

INCOSE  Pg  4.12-13 

100%  -  100% 
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